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Introduction 
and Overview 

rfi 

JL. his book has many dimensions. It provides a conceptual scaf- 
folding and practical guide that evaluators, curriculum supervisors, school 
administrators, policy makers, and teachers can consult as they respond to 
the growing challenges of cultural diversity in schools. It introduces ways 
to assess the capacities of various professional development programs, 
curriculum transformation projects, diversity initiatives, and study abroad 
opportunities to realize complex cultural objectives. It serves as a cultural 
studies manual that can assist teachers and school administrators to treat 
culture much as they treat history, geography, or literature — as a curricu- 
lum basic with identifiable academic content, intellectual utility, skill- 
building capacity, and civic power. Most centrally, however, it introduces a 
content-focused form of program evaluation that indexes program quality 
to the culture-revealing features and effects of goals and objectives, pro- 
gram designs, learning opportunities, and program outcomes. 

The book contains three parts, each of which brings particular 
elements of cultural education program evaluation into view. Individually 
and collectively, each section strives to peel apart the elements of cultural 
education programs to reveal a rich array of indices by which to evaluate 
the many dimensions and possibilities of cultural studies programs. Part I 
emphasizes the input side of programs by focusing on various conceptual 
and programmatic dilemmas that face evaluators as they seek to assess the 
quality of cultural education opportunities being made available to pro- 
gram participants. Chapter 1, an introductory essay, focuses on the 
problematics of definition that center around concepts of culture and 
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cultural education. It identifies the elements of a culturally rich, educa- 
tionally evocative concept of cultural education and defines four dimen- 
sions of cultural education programs that warrant the attention of evalua- 
tors. Chapter 2 focuses on the dilemmas of evaluating program content 
and introduces a cultural education learning opportunity index that may 
help evaluators identify the content of programs. In addition, it offers a 
conception of cultural education learning that may enable evaluators to 
monitor the evolution of cultural learning among participants as a program 
develops over time. Last in this first section, chapter 3 focuses on the 
evaluation of program leaders by laying out ways to assess how leaders 
define culture and cultural education, how they organize opportunities for 
cultural teaching and learning, and how they engage participants in cul- 
tural reflection and network-building. 

Part II, comprising chapters 4 and 5, addresses the evaluation of 
program outcomes, suggesting ways to identify, index, and assess transfor- 
mations in the professional behavior of participants on one hand and the 
quality of instructional materials on the other. Unlike more traditional 
approaches in the field of cultural teaching and learning which define 
outcomes in terms of transformations in attitudes and adaptability, we take 
a different approach here by linking assessments of program quality to 
concrete, analyzable, and/or measurable behavioral or performance out- 
comes among participants, whether they are classroom teachers, staff 
development leaders, curriculum supervisors, or school administrators. 
Chapter 4 presents ways to index the effects of programs to the evolving 
capacities of participants to enhance cultural teaching and learning in 
classrooms, pre-service teacher education, staff development, and continu- 
ing education programs. In addition, the chapter defines program out- 
comes in relationship to participant engagement in professional societies 
and community life. Chapter 5 presents strategies for the assessment of 
program outcomes as measured by the characteristics of instructional 
materials made available to learners — an often unevalualed outcome. 

Finally, Part III of the book focuses on the relationship of program 
inputs and outputs simultaneously, attending to two themes that we 
believe to be of compelling importance for evaluators. Chapter 6 centers 
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on the economic dimensions of cultural education program evaluation by 
presenting ways to reveal the full range of resource commitments that 
teachers, students, administrators, and others must make in order to 
support a cultural education program, assess its utility, and/or improve its 
educational capacity. Chapter 7 revisits the issue of cultural bias in the 
assessments of students, focusing on the need for new performance-based 
assessment and on the dilemmas of accomplishing it well. 

The contents of the book rest on a series of assumptions about the 
evaluation of cultural education programs. First, there is an assumption 
that effective cultural education programs promote specific kinds of goals: 
a complex view of cultural possibility, an awareness of the cultural habits of 
program participants or learners, a recognition of diversity within cultures, 
a realization of the cultural complexities and hierarchies in classroom 
settings and in instructional materials, and an awareness of the dilemmas of 
intcrcultural communication within education settings, for example. 
Second, there is an assumption that cultural study and intcrcultural experi- 
ence constitute twin pillars on which effective professional development 
programs rest. Third, there is an assumption that content-rich programs of 
cultural study and experience arc indispensable elements in the continuing 
education of teachers and school administrators who have typically re- 
ceived their training in culturally-encapsulated enclaves and are thus 
unprepared to deal with culturally complex educational environments. 
Fourth, there is an assumption that the cultivation of critical reflection 
about cultural repertoires requires programs that foster encounters be- 
tween people and groups who possess different perspectives. Fifth, there is 
an assumption that systematic opportunities for rigorous study, observa- 
tion, reflection, and immersion in new cultural habitats are indispensable 
aspects of effective cultural education, providing new ways for educators to 
revisit, reflect, and reconstruct their own cultural and educational practices 
and, in the process, help colleagues and students to do so as well. Sixth, 
there is an assumption that community building, curriculum development, 
and transformations of education practice constitute important outcomes 
of effective cultural education programs — more important even than the 
more traditionally-defined outcomes such as personal satisfaction, indi- 
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vidua! psychological development, or advancement on an intercultural 
development scale. Finally, this book rests on the assumption that cultural 
education program evaluators — or, better yet, teams of evaluators — need 
to have expertise in cultural analysis, qualitative and quantitative methods 
of data collection and analysis, and various forms and functions of program 
evaluation. Only when they are able to draw on this breadth of expertise 
can evaluators both envision and examine possibilities for designing and 
conducting cultural program evaluations that advance the purposes and the 
powers of the programs. 



Barbara Finkelstein 
University of Maryland, College Park 
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T 

.JLhis chapter presents an approach to the design 
and evaluation of cultural education programs that 
will assist program leaders and evaluators to anticipate 
the array of dilemmas that the assessment of cultural education programs 
inevitably presents. First, there are dilemmas of definition that center 
around concepts of culture and cultural education. Second, there are the 
problematics of identifying program features that are culturally specific. 
Third, there are the challenges of defining indices of cultural education 
program excellence that will guide the assessment of program quality. And 
finally, linked to the third, there are the trials of constituting culturally 
sensitive, assessable outcome measures. 

In this chapter, we address each of these dilemmas in turn. We 
approach problems of definition by outlining the concepts of culture and 
cultural education that have dominated the five curriculum domains 
within which cultural studies and/or programs have been traditionally 
lodged: (a) ethnic studies, (b) area studies, (c) global studies, (d) cross- 
cultural studies, and (e) studies called multicultural. Second, we discuss 
four dimensions of cultural education program excellence that warrant the 
attention of evaluators: the vision of leaders, the knowledge dimensions of 
program contents, the instructionally evocative and communicative power 
of programs, and the capacity of programs to cultivate cross-cultural 
connections and transcultural communities. Finally, we explore ways to 
index the outcomes of cultural education programs. 
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The Meaning of Culture 

Different definitions of culture have evolved over time, acquiring 
meaning and focus as serial generations of anthropologists, social psycholo- 
gists, sociolinguists, and education scholars have explored the habits of 
heart, mind, and association that both distinguish one group from another 
and define the forms that education has taken. By culture, we are referring 
to historically-constructed patterns of associations and meaning, or tem- 
plates of experience, to use Clifford Geertz’s (1973) concept, that are 
actively transmitted from one generation to the next through an array of 
symbolic forms, social habits, material constructions, and educational 
efforts. 

Indeed, culture is, by definition, educationally constituted and 
transmitted. Culture is embedded and conveyed in the languages we use, 
the stories that we tell, the myths that we cherish, and the poetry we 
create. It is communicated socially, through daily habits of association, 
rituals of performance, patterns of speech, and rites of passage. It is also 
revealed and transmitted materially in architectural remnants, physical 
objects, and artistic forms. No matter what its expression, culture consti- 
tutes the forms by which people communicate, perpetuate, and develop 
their knowledge and understanding of life and their relationships to one 
another. 

Culture is elusive and hidden, commonly outside the margins of 
individual awareness, and embedded in habits that are typically taken for 
granted. Its elusive quality has led certain cross-cultural theorists to liken it 
to an iceberg, with only its peaks revealed (Weaver, 1993, 1994). But this 
view of culture obscures its fluid and dynamic qualities. Indeed, culture, as 
we regard it, is vibrant and animated, actively constructed, reproduced, 
and sometimes transformed in the course of daily life — especially in cul- 
tural education programs. What with the elusive and shifting qualities of 
culture, its study takes on the qualities of educational detective work. It 
requires teachers and learners together to engage in creative forms of social 
snooping through which they can ferret out implicit meanings, discover 
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the previously inaccessible, and learn more about their own culture and 

the cultures of the students whom they teach (Bateson; 1994; Douglas; 

1975; Hall & Hall, 1990). 

In summary, cultural education, as we use the term, refers to pro- 
grams that cultivate cultural vision and knowledge, communicative dispo- 
sitions, community-building skills, and cultural literacy as important 
educational objectives. It refers to a content emphasis that includes cul- 
ture-specific knowledge, cultivates perspective-consciousness, and pro- 
motes organized opportunities for participants to compare, reflect, and 
decode meaning. Cultural education also refers to program processes that 
enable participants to enter into productive exchanges with people in 
possession of culturally different habits and ways of knowing, being, and 
educating. In addition, it refers to programs that assist educators to serve 
effectively as cultural message-carriers, thereby helping students to con- 
nect their own experiences and ways of knowing to those of others, to 
giaft new concepts onto familiar ideas, and to seek out both an under- 
standing and appreciation of multiple perspectives. Further, cultural 
education projects a vision of educational practices that constructs respect- 
ful social relations between and among culturally diverse students. Most 
centrally, our concept of cultural education engages participants in system- 
atic explorations of the phenomena of culture. 



The Terrain of Cultural Education 

The terrain of cultural education is dazzlingly complex, an area of 
inquiry without fixed boundaries, parameters, terminologies, anchoring 
concepts, or categories of understanding. As theory and as practice, cul- 
tural education has lodged and flourished in diverse intellectual and social 
enterprises (Banks, 1993; Grant, 1993; Paige, 1993; Seelye, 1994). One 
site for the cultivation of cultural education has been area studies pro- 
grams — educational units that center on inves gations of particular regions 
or nations of the world; define language proficiency as an important 
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outcome; and engage participants in culture-specific, specialized, relatively 
in-depth studies ofliterature, history, and political and social life. Area. _ _ 
studies programs commonly invite students to acquire deep knowledge of 
a language other than their own. Despite encouraging the study of culture, 
however, area studies directors have, until recently, typically proceeded on 
the assumption that the study of culture is so inextricably tied to language 
learning that cultural learning is an inevitable outcome of language mas- 
tery, rather than a subject to be explored in its own right (Duranti & 
Goodwin, 1992). 

Cultural education has been approached in a different way in global, 
world culture, and international studies programs. These specialties privi- 
lege the study of representative nations and cultures from around the 
world, seeking to familiarize students with geographic, historic, and social 
matters of fact, with an intent to cultivate global understanding, interna- 
tional consciousness, and a sense of place and connection. Matters of 
perspective from within the various nations and regions of the world are 
commonly less important than geographic and cultural distinctions. 

Cultural education has acquired a different definition and purpose in 
the field of cross-cultural studies where program leaders frame the central 
problems around the rigors of living abroad and link program goals to the 
cultivation of culturally and psy Jogically adaptive skills, relying on 
psychologists and anthropologists for guidance and wisdom (Hall, 1959/ 
1981, 1966/1982, 1976/1981; Hall & Hall, 1990; Paige, 1993; Seelye, 

1994). Cross-cultural studies leaders, unlike their counterparts in interna- 
tional studies programs, are less focused on matters of fact than on matters 
of personal adjustment. 

More recently, multicultural educators have linked cultural educa- 
tion to the cause of cultural diversity in the United States. Constructing 
concepts of multicultural teaching and learning that define cultural studies 
as forms of political and social education, they have focused attention on 
minority cultures; developed strategies to assure dignity, competence, and 
voice for all students; and attempted to combat the baleful effects of ethno- 
centricity, racial prejudice, negative stereotyping, and discrimination 
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(Banks, 1993; Grant, 1993; Hawley & Jackson, 1995; Paige, 1993). In a 
slightly different mode, advocates of ethnic studies have also responded to - 
the challenges and opportunities of diversity as they seek to recover the 
voices of previously invisible groups and, through this means, enlarge the 
intellectual reach of history, literature, and related academic disciplines 
(Awkward, 1995; Banks, 1993; Takaki, 1993, 1994). 

Our understanding of cultural education borrows prodigiously from 
each of these traditions, yet also imports an array of definitions and inter- 
viewing and observational techniques from the fields of oral history and 
anthropology, the parent discipline of cultural studies. As conceptualized 
in this book, cultural education is a form of area studies that emphasizes 
in-depth study of particular cultures as an effective way to understand 
alternative habits of heart, mind, and association (Douglas, 1975; Geertz, 
1973; Hall Sc Hall, 1990). It is a form of global or international education 
that emphasizes the value of comparative cultural study as a way to culti- 
vate an awareness of place and connection and to develop a capacity for 
cultural reflection and mutual understanding. In addition, cultural educa- 
tion is a form of multicultural teaching that encourages studies of alterna- 
tive cultures, voices, and communication styles; celebrates diversity; and 
links cultural studies to school improvement and new civic possibilities. In 
addition, cultural education refers to programs that assist educators to 
function effectively and reflectively as cultural message-carriers, helping 
students to understand other cultures and alternative ways of knowing. 

The design and evaluation of cultural education programs with these 
levels of complexity are not simple tasks. Program leaders and evaluators 
alike will have to navigate their way through a maze of definitional and 
programmatic dilemmas that inevitably await them as they organize and 
assess approaches to the challenges and promises of diversity, frame and 
analyze educational goals and objectives, assess and choose among alterna- 
tive curriculum contents, evaluate instructional materials and cultural 
encounters, cultivate culturally sensitive forms of evaluation and assess- 
ment, and, finally, constitute ways to monitor, evaluate, and refine programs. 
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What follows in the next section is an overview of four dimensions 
of .cultural. education that .warrant-consideration as an evaluator gathers” 
information and generates strategies to assess cultural education programs. 



Four Dimensions of Cultural 
Education Programs 

In our view, there are four features of cultural education programs 
that, taken together, can be used as a template to explore the content of 
cultural education programs. First, there is a visionary dimension, which 
involves evaluators in an exploration of leaders’ and participants’ concep- 
tions of culture, images of community, and definitions of educational 
possibility. Second, there is a knowledge or information dimension, 
which requires evaluators to explore and assess the intellectual power of a 
program including its culture-revealing, instructionally evocative, and 
cultural skill-building qualities. Third, there is a communicative dimen- 
sion in cultural education programs, which challenges evaluators to assess 
the capacity of leaders and participants to articulate clearly visions of 
culture and community; cultivate cultural skills; communicate across 
boundaries of race, class, and gender; integrate alternative points of view; 
and create cross-cultural learning occasions. Finally, there is a commu- 
nity-building dimension to cultural education programs, which, if 
creatively analyzed, reveals the intercultural reach and network-construct- 
ing capability of a program. In the following section of this chapter, we will 
elaborate on each of these dimensions of cultural education programs. 

Vision 

Consistent with the view that education leaders, like all other hu- 
mans, import concepts of culture into their work, it is necessary for an 
evaluator to analyze the definitions of culture, approaches to diversity, and 
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cultural expectations that are encoded in statements of program goals, 
^objectives, expectations, and other communications. Savvy program 
evaluators might view goals and objectives as communicative texts that 
reveal notions of culture, concepts of cultural education, and assumptions 
about cultural teach ng and learning. As evaluators sort through the sub- 
stance of goal statements, they can discover if a leader or a program partici- 
pant approaches cultural education with a relatively narrow or broad point 
of view. For example, some goal statements nest the study of particular 
cultures and societies within larger cross-cultural themes and thus ground 
cultural study in transcultural concerns and broad curriculum frameworks 
(Eder, 1994/1995; Finkelstein, 1992; Finkelstein & Eder, in press). Other 
goal statements might define purposes primarily in relation to the acquisi- 
tion of language proficiency, thereby suggesting an implicit notion of 
cultural learning as a language-acquiring act (Vasquez, Pease- Alvarez, & 
Shannon, 1994). Similarly, goal statements that emphasize the study of 
minority'’ group cultures and respect for minority' group members, linking 
them to issues of language learning or studies of particular cultures, suggest 
that a program leader defines the importance of cultural study in relation 
to issues of status, identity, and cultural politics and defines culture itself as 
a form of political expression (Awkward, 1995; Banks, 1993; Hawley & 
Jackson, 1995; Nieto, 1992). No matter what the substance of the goals 
and objectives, their cultural content will reveal the particularities of a 
leader’s program vision and provide important sources of information for 
evaluators. 

Since it is not uncommon in cultural education programs to have 
complex interdisciplinary and/or cross-cultural leadership teams with 
specialized skills, interests, and expertise, evaluators can often discover in 
the same program several visions and points of view regarding the meaning 
of culture and of cultural education. Taken both singly and collectively, the 
voices of leaders describing their hopes for the program, the choices they 
make, and the educational approaches they choose will, if carefully re- 
corded, constitute cultural vision statements of uncommon utility for 
those concerned with program evaluation. 
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Yet another approach to analyzing the vision dimension is to explore 
the criteria that program leaders use to define a focus for their programs. 
This is not a simple matter, since program organizers are often unaware of 
the totality of criteria that they use. Those who define culture generically 
to include the study of any or all cultures without a central focus on any 
particular cultures are typically unable to cultivate program features of 
sufficientdepth to enable participants to acquire perspective-consciousness 
in systematic and reflective ways. If, on the other hand, the program 
leaders identify content that engages participants exclusively in the study of 
a single culture, without reference to larger cross-cultural themes or 
culturally comparable elements, then the value of the program for curricu- 
lum development and instructional improvement across the curriculum 
will be diminished. An alert evaluator can be on the lookout for the educa- 
tional utility of program elements that catapult participants outside the 
realm of their ordinary experience and require them to struggle together 
to understand and compare habits of heart, mind, and association and, as a 
result, constitute new ways of incorporating cultural study into their 
practices. 

Still another approach to discovering the visionary reach of a cultural 
education program is to explore participants’ concepts of culture and 
cultural education, not only as they are revealed as program outcomes, but 
also as they might shape their expectations of what can and should be 
learned in a cultural education program. 

A final way for evaluators to explore the visionary dimension of 
programs and leaders is to pay close attention to the structure and se- 
quence of program elements by analyzing the relationships among parts, 
noting thematic elements, and attending to omissions. By analyzing the 
choice of program emphases, readings, excursions, and cultural encoun- 
ters, evaluators can assess leaders’ and participants’ understanding of the 
subject matter, their sensitivity to the cultural perspectives of participants, 
and their grasp of participants’ roles as bearers and mediators of culture. 
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Knowledge 

A second feature of cultural education programs that warrants the 
attention of evaluators is the knowledge or information dimension. The 
analysis and assessment of program content is not an easy task, since, as we 
have seen, the contents of cultural education programs can vary, the 
sources of cultural information are complex and dispersed, the availability 
of cultural informants is problematic, the opportunities for cultural en- 
counters vary from one community to another, and the cultural habits of 
program leaders and participants commonly differ. What is more, culture, 
as we have discussed, is not simply apprehended, nor is its precise content 
easily fixed or sequentially developed and learned. Thus, it is important for 
an evaluator to generate knowledge-centered evaluation strategies that will 
reveal the content of a cultural education program and assess the kinds of 
learning opportunities that a program presents. Knowledge-centered 
evaluations can reveal what is taught about a culture and what is not, 
which features of a culture come into view and which do not, and which 
perspectives from outside and inside a culture are available for reflection 
and observation and which are not. Most important, knowledge-centered 
evaluation can identify the breadth, depth, and range of specific cultural 
education program content by arraying the various program elements 
along a continuum of culture-revealing learning possibilities. 

At one end of the continuum are opportunities to study culture 
vicariously by examining a book, film, textbook, scholarly article, or 
journalist’s report. These cultural artifacts might reflect the views of 
cultural outsiders or cultural insiders. No matter which, each is the w r ork 
of a cultural mediator who filters culture at the same time, he or she reveals 
it. For a program leader, participant, or evaluator, it is as important to 
identify the perspectives of these cultural mediators as it is to learn the 
substance of their cultural messages. 

Aligned midway along the cultural education learning continuum arc 
opportunities to study culture directly through personal encounters with 
cultural insiders, hands-on exploration of authentic objects, or firsthand 
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observations of cultural performances such as dramatic productions or 
speech acts. Direct encounters reveal culture in evocative ways; since 
cultural insiders enact culture and reveal its forms as well as impart infor- 
mation about it. Direct encounters also require acts of cultural improvisa- 
tion as people in possession of different ways of knowing and associating 
struggle to understand one another, to communicate in culturally specific 
ways, and to enact habits that are amenable to study, observation, and 
reflection. 

At the far end of the cultural education learning continuum are 
opportunities to study culture experientially through total immersion in an 
alternative cultural setting — alone, as it were, in the company of cultural 
strangers. When program participants take opportunities to live, learn, and 
communicate in unmediated settings on a daily basis, they can observe, 
feel, and experience cultural complexities that are otherwise invisible or 
impenetrable. When they dwell for a time in the experience of others, they 
can acquire multiple perspectives from within a culture and, if they have 
the opportunity to reflect on their experience, learn to re-experience their 
own culture as well. 

These progressive points along the cultural learning continuum can 
represent successively deep levels for acquiring knowledge. Evaluators, as 
we have seen, can determine the depth of cultural education offered by 
analyzing the kinds of cultural opportunities that are made available. More 
complex programs incorporate all three cultural learning inodes: the 
mediated, the direct, and the totally immersed. Less complex ones may 
include only the vicarious or only the experiential. In addition, evaluators 
can distinguish high quality cultural learning opportunities from those that 
are not by attending to the multiplicity of perspectives that are made 
available to participants within each learning mode. Indeed, content-rich, 
rigorous opportunities potentially occur when participants have the occa- 
sion to discover diverse points of view or perspectives from within a 
culture; study, observe, reflect on, and experience alternative habits of 
heart, mind', and association; analyze similarities and differences between 
and within cultures; and critically analyze images of the target culture in 
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instructional materials, lesson plans, films, and objects and thus transcend 
stereotypes. 1 . . ... . • .... 



Communicative Disposition 

A third feature of cultural education programs that justifies the 
attention of evaluators is the communicative dimension. By communica- 
tive dimension, we refer to those aspects of cultural education programs 
and program leadership that reveal the intercultural communication skills 
of leaders and structure opportunities for participants to: 

■ decode what they see, reflect on their own cultural habits, 
suspend judgment, and learn to listen, hear, and observe; 

■ communicate across boundaries of race, gender, and class; 

H cultivate inter-group sensitivity; and 

■ articulate concepts of culture and community without 
overgeneralizing or becoming judgmental. 

The importance of communicative disposition can hardly be overes- 
timated, since, as we have seen, high quality cultural education programs 
will invariably require the help of cultural strangers in possession of various 
ways of communicating, speaking, associating, teaching, and transmitting 
information. Thus, it is important for an evaluator to assess the ability of 
program leaders and participants to identify and prepare appropriate 
cultural informants, to decode meaning or “read" cultural messages atten- 
tively, and to become aware of communication obstacles including their 
own cthnocentricities and communicative styles. 

For an evaluator, the exploration of communicative disposition 
requires the identification and assessment of opportunities for participants 
to study communication habits among groups within the target culture; to 



'The concept of stereotyping is complex t since it is impossible to say much about a culture without generaliz- 
ing. In our view, stereotyping is an extreme form of generalizing that obscures complexities of culture. 
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practice communication skills by writing letters, making phone calls, 
constructing E-mail correspondences, etc.; to reflect on a program leader’s 
approach to communication; and to engage vicariously in role playing 
situations or in simulations that mimic real-life possibilities, In addition, 
evaluators should examine programs to look for occasions in which par- 
ticipants can decode what they see, reflect on their own cultural habits, 
cultivate skills of listening and observing, and suspend judgment. 

Evaluators can also assess the communicative features of programs by 
attending to the number of cultural contact zones made available to par- 
ticipants and by measuring the number and intensity of communication 
points within these enclaves ofintercultural reflection and perspective - 
taking (Finkelstcin, 1996). 

Community Building 

Finally, there is a community-building feature in cultural education 
programs which, if creatively analyzed, will reveal the intercultural reach 
and network-constructing capability of a program. By community build- 
ing, we mean the opportunities presented to leaders and participants alike 
to cultivate new communities of understanding that will help them iden- 
tify themselves as a distinct cultural group, build and/or expand networks 
of cultural informants, construct occasions for new forms of community- 
building, and cultivate opportunities for sustained cross-cultural collabora- 
tion on educational issues of mutual concern. 

For evaluators, the exploration of the community-building features 
of a program is not simple since it will inevitably involve them in assess- 
ments of pre-existing networks, whether they are networks of teachers 
sharing cultural information and instructional resources, program leaders 
collaborating with groups of cultural informants, or educators and scholars 
engaging in joint research projects or cultural exchanges. Evaluators will 
need to assess what, if any, new networks of collaboration have evolved as a 
result of a program; what new resources have become available for partici- 
pants; and what cultural bridges, if any, have been built as a result of 
program opportunities and processes. 
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When they are taken together, the four dimensions of cultural 
education — vision, knowledge, communicative disposition, and commu- - 
nity building — form an evaluative structure that program evaluators might 
consider as they seek to uncover and assess the breadth and depth of 
cultural education programs in terms of their leaders, program contents 
and program outcomes. 



Evaluating Outcomes 

The role of the cultural education program evaluator goes beyond 
the discovery and assessment of the vision of program leaders and the 
quality of program contents. It includes, as well, an assessment of the 
effects of both the leaders and the contents on the behavioral repertoires 
and attitudes of program participants. As one might expect, this is a com- 
plex area for analysis since program participants are as likely to be school 
administrators, curriculum supervisors, or policy makers as they are to be 
classroom teachers. Each of these educational actors works through differ- 
ent forums and each will manifest his or her own cultural knowledge in 
different ways. It is important, therefore, for an evaluator to pay close 
attention to the decision-making terrains of program leaders and to iden- 
tify appropriate outcome indices. For a classroom teacher or curriculum 
supervisor, for example, the relative success or failure of a cultural educa- 
tion program might be indexed to his or her ability to frame cultural goals; 
to filter information and choose content; to create academically sound, 
culturally sensitive instructional materials; to identify appropriate cultural 
informants and encounters; to generate culturally sensitive measures of 
student achievement; and to cultivate new communities of understanding 
among professionals and/or students. For a policy maker, on the other 
hand, the relative success of a program might be registered in his or her 
ability to construct rules and regulations that reflect a modicum of sensi- 
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tivity to the contents and possibilities of cultural education; to assess the 
strengths and weaknesses of different approaches to cultural education as 
he or she deploys resources and assesses risks and opportunities; and to 
identify the kinds of programs that link cultural study to school improvement. 

The challenge of locating ways to measure the relative contributions 
of program leadership and design on program outcomes is especially 
awesome for cultural education program evaluators. It is difficult, if not 
impossible, to distinguish the effect of different features of programs on 
participants without richly nuanced participant data. Culturally-adept 
evaluators will thus construct information-gathering strategies that invite 
participants to reveal critical moments in the evolution of their behavior, 
attitudes, and perceptions. 

Evaluators have no cause to congratulate themselves if they measure 
outcomes only by comparing the perceptions of program leaders and 
participants in a kind of global appraisal derived from a continuum of 
responses on a limited Likert-like scale. In our view, it is possible to ap- 
proach the task more creatively and rigorously, through the discovery of 
ways to define the relationship of culturally sophisticated concepts and 
skills to concrete social, intellectual, pedagogical, or civic behavioral 
outcomes. For example, in the case of curriculum supervisors, evaluators 
might develop standards for cultural knowledge, communicative disposi- 
tion, and community building that can be used to measure how well 
supervisors frame cultural goals and content emphases, select instructional 
materials, identify cultural advisors, and otherwise prepare and conduct 
professional development opportunities. In the case of teachers, evaluators 
might develop criteria with which to evaluate curriculum materials and 
lesson plans. For policy makers, evaluators might construct frameworks for 
assessing the costs, benefits, and effectiveness of different cultural educa- 
tion programs. 
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Conclusion 

When evaluators attend systematically to'the exploration of cultural 
concepts as they appear in particular cultural education programs, they can 
acquire sufficient information from which to analyze and evaluate the 
strengths and weaknesses of each one. Evaluators can do so, as we have 
seen, if they attend to four indispensable components of cultural education 
programs: (a) a vision and concept of culture, (b) knowledge, (c) commu- 
nicative disposition, and (d) community building. Subsequent chapters, 
following this category structure, will suggest research questions that can 
help evaluators assess the qualities and features of effective cultural educa- 
tion initiatives. 
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TSJ 

X ^ o matter how coherently planned, sensibly 
constituted, or well led they may be, cultural educa- 
tion programs take on a life of their own when they 
are enacted. To reveal and assess the quality of programs, evaluators need to 
find ways to explore and index the relative capacities of various program 
elements to cultivate vision, reveal culture, raise perspective consciousness, 
and provide opportunities for cultural border crossing, intercultural com- 
munication, and reflections on teaching and learning practices among 
participants. 

The identification of program-specific effects alone poses a range of 
potential dilemmas. There is, of course, the traditional issue of disentan- 
gling the effects of program content from other programmatic and non- 
programmatic influences. Typically, these include the quality of cultural 
education leadership, the image-making power of mass media, the relative 
influence of early cultural learning, and the cultural repertoire and images 
of the target culture with which participants enter into any program. An 
evaluator will need to understand the ways in which program materials 
contradict and/or complement other forms of information that participants 
might have acquired in advance of their participation in the program. 

Beyond the problematics of identifying program-specific effects, 
there is yet another dilemma that presents itself. In cultural education 
programs, both program elements and program outcomes are nonlinear, 
multidimensional, discontinuous, sustained, and revealed in oftentimes 
unevaluated spaces, such as classrooms, communicative repertoires, profes- 
sional societies, and instructional materials. Thus, it is necessary for an 
evaluator to take special care to disaggregate and specifically identify the 
various program elements and to construct ways to assess how these 
program elements might determine outcomes. 
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As if this were not enough, evaluators also must determine which 
aspects of programs to evaluate, using indexes of program excellence that 
are sensitive to important features of high-quality cultural education 
programs such as those introduced in chapter 1: (a) the vision-defining 
quality of the program, or opportunities for systematic cultural exploration 
and interpretations of cultural meaning and experience; (b) the value of 
the content or knowledge as determined by thematic elements that reveal 
and/or obscure important features of a culture, readings and encounters 
that expose or conceal multiple perspectives from within a culture, and 
opportunities that invite participants to discover similarities and differences 
between and within cultures; (c) the instructionally evocative power of the 
program, that is, its capacity to assist participants in translating cultural 
knowledge and experience into effective instructional materials, new 
curricula, and new frames of cultural reference and professional expres- 
sion; and (d) the community-building features of the program. 

Beyond the identification of assessable program features, cultural 
education program evaluators also need to disentangle an extraordinarily 
complicated learning process. Needless to say, evaluators must have suffi- 
ciently versatile methodological repertoires to generate sensitive and 
appropriate program data. We are projecting a model of evaluation that 
defines the features of cultural education programs to be evaluated (leader- 
ship, programs, and outcomes) and identifies phases in the evolution of 
cultural learning. This approach to evaluation provides an analytic scaffold- 
ing or conceptual framework through which evaluators can discover the 
anatomy of a cultural education program. 

Using this kind of model, evaluators can assess relationships between 
program elements and program outcomes in two ways: formatively, 
through various phases of learning as a program proceeds, and 
summatively, through retrospective participant assessments and analyses of 
the cultural repertoires of participants after the program is completed. 

Following is a set of conceptual tools and methodological repertoires 
for generating data and using it to assess each program feature and stage of 
learning. This form of data collection will enable evaluators to analyze and 
disaggregate each program feature vertically through all stages of cultural 
learning and longitudinally after the program is completed. 




PART I: Designing, Defining, and Evaluating Cultural Education Programs: The Input Side 



Five Phases of Cultural Learning 

~WhaL follows' is ariexplanationof ways' to evaluate programs wi thin 
and across each phase of cultural learning. The challenge for an evaluator is 
to discover assessable features of a program, index them to the developing 
cultural repertoires of participants, and locate ways to observe the follow- 
ing phases of their evolving concepts of culture and approaches to cultural 
difference. 

First, there is a phase of cultural imagining when participants analyze 
program descriptions, prepare to participate in a cultural education pro- 
gram, imagine the promise and possibility of participation, and otherwise 
frame their expectations. For an evaluator, this is the time to establish a 
pre-program understanding of participants 5 cultural repertoires and to 
determine participants 5 images of the target culture, their interpretations of 
program materials, the instructional modifications they might hope to 
make, and their views of culture and cultural education. 

Second, there is a phase of mediated cultural learning when partici- 
pants read, participate in seminars, discover community resources, and 
engage in the study of instructional materials. The challenge for an evalua- 
tor here is to discover what features of the culture are obscured or revealed 
for the participants; to analyze the evolving capacities of participants to 
identify the nature of cultural filtering by scholars, film producers, text- 
book writers, poets, program leaders, etc.; and otherwise to explore the 
uses which participants are making of information sources. 

Third, there is a phase of cultural encounter when participants 
become immersed in a culture and learn systematically to observe, de- 
scribe, analyze events, anticipate dissonances, and learn from informants 
directly. During this phase of cultural immersion, participants engage 
informants and live, study, and/or work directly with them, without 
mediation from other authorities or agencies and without opportunities 
for immediate debriefing. During this phase of cultural learning, partici- 
pants test their powers of observation, their communication skills, and 
their abilities to discover alternative perspectives in culturally different and / 
or alternative community or educational settings. For an evaluator, this 
third phase of cultural learning requires the cultivation of information that 
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will reveal the communicative repertoires of the participants as embodied 
in their abilities to: learn the perspective of a cultural other; frame similari- 
ties and differences in their approaches to the content at hand; construct 
opportunities for deeper dialogues; imagine and build new forms of com- 
munity collaboratively; and acquire previously inaccessible instructional 
materials for students, colleagues, resource centers, etc. 

Fourth, there is the phase of retrospective cultural reflection when 
participants look backward to revisit the whole of their experience and 
make plans to apply the knowledge they have acquired in an array of 
educational settings. During this stage, evaluators need to generate infor- 
mation that discloses changes, if any, in the participants’ images of the 
target culture, in their perspectives from within it, in the resources that 
they identify to reveal it, and in their ideas concerning the translation of 
cultural knowledge into new forms of cultural teaching and professional 
expression. 

Fifth, there is a phase of cultural application in which participants 
apply their cultural knowledge to the preparation of instructional materi- 
als, creation of cultural encounters, design of modes of assessment, enact- 
ment of curriculum transformations, etc. For an evaluator, the assessment 
of each of these applications will reveal the effects of programs on the 
depth of participants’ understanding, cultural sensitivity, and knowledge, 
once a program is completed. 

The analysis of each of these programmatic phases provides evalua- 
tors with sufficient information to make informed programmatic assess- 
ments both formatively, as a program is being enacted, and summatively, 
when it is completed. While programs may vary in depth and scope, the 
analysis of each program phase is no less fruitful for one-day programs than 
it is for extended seminars with study abroad components. Still, evaluators, 
in our view, need to distinguish between programs of short and long 
duration in two ways: on the input side, they must explore what kinds of 
learning opportunities are available, and on the output side, they must 
assess the depth and type of change in participants’ conceptions of cultural 
programs and in their behavioral and communicative repertoires. When 
each feature of a cultural education program is systematically evaluated, it 
is possible to discern the depth, breadth, and development of programs, 
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not only as leaders intend them, but also as participants develop cultural 
capacities within them. . . . . 

What follows is an element-by-elemeiit analysis of ways to generate 
information about various emphases and phases of a cultural education 
program. We should note that an evaluator will need to explore the rela- 
tive emphasis of each feature in any single program, bearing in mind that 
high-quality programs integrate all of the features but that typical programs 
often engage participants in academic study exclusively, without cultural 
encounters or reflective discussion. 



Cultural Imagining: The Discovery of 
Vision, Knowledge, Communicative 
Disposition, and Community Engagement 

The assessment of this initial phase involves evaluators in systematic 
efforts to elicit four kinds of information in advance of a program: (a) the 
substance of participants’ cultural vision, such as their images of the target 
culture, cultural sensitivity, reflective capacities, and professional reper- 
toires; (b) the quality of participant knowledge, or their familiarity with 
scholarly literatures, instructional materials, and community resources, as 
well as the relative awareness that they reveal of the complexities of cult ure 
and the dangers of overgeneralization; (c) the levels of communicative 
competence among participants as reflected in the strategies of cultural 
teaching and learning that participants deploy when they teach about other 
cultures generally and the target culture specifically; and (d) the concepts 
of community and community-building as they are revealed in profes- 
sional repertoires that participants hope to acquire as a result of their 
involvement in the program. 

Taken together, these four areas of discovery will enable evaluators to 
cultivate a cultural education profile for each participant, assessing the 
condition of their knowledge and the quality of their cultural, reflective, 
and communicative capacities before a program begins. 
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The One-day Workshop: Questionnaires 

Not unexpectedly, evaluators will need to engage in acts of analytic 
improvisation in order to construct evaluation strategies that are appropri- 
ate for the dazzling diversity of programs that constitute cultural education 
initiatives in the United States. In one-day programs the best an evaluator 
can do is to construct mini-questionnaires that can be answered quickly 
without detracting from the substance of the program. For a one-day 
workshop entitled “Discovering the Invisible: Studyingjapanese Culture 
and Education through Objects,” held at the University of Maryland in 
College Park in the fall of 1995, for example, the evaluator constructed a 
“cultural probe” in an attempt to assess the relative complexity of partici- 
pant teachers’ notions of cultural teaching and learning: 

What level do you teach? O K— 3 O 4-8 □ 9—12 O Other 

1 . What do you hope to learn about culture in this workshop? 

2. What have you emphasized about Japan in your teaching? 

3. What resources have you been using and are you satisfied with 
them? 

4. Have you had any contact with people from Japan? If so, please 
describe. 

5. To me the most important thing to remember when teaching 
about another culture is . . . 

6. How do you intend to use, share, and disseminate the knowledge 
you will take from this workshop? 

These questions served as a kind of cultural sophistication index 
through which the evaluator was able to assess the complexity of cultural 
knowledge, instructional resources, intercultural experience, and cultural 
sensitivity that participants brought to the program. If, as it happened, a 
participant emphasized the importance of using maps and globes, had used 
the portrayal in a world cultures text, and was unaware of local or national 
resources and the cultural dilemmas of defining place and location, then it 
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would be fair to conclude that he or she was in an early stage of cultural 
learning with undeveloped capacities to analyze information critically, 
discover multiple perspectives from both outside and inside a culture, 
reflect on the filtered quality of information, or seek out cultural infor- 
mants and subtle information sources. 



Programs of Longer Duration: Questionnaires as 
Reflections of Cultural Thinking 

The process of pre-program cultural probing can be more nuanced 
and sophisticated in programs of longer duration, in which participants 
have opportunities for extended study and experience and have time and 
motivation to integrate cultural learning into new forms of educational 
practice. The evaluation of several Japan-related programs of the National 
Intercultural Education Leadership Institute, conducted from 1990 to 
1994, illustrates an array of methods used to generate information and 
reveal the cultural education capabilities and interests of participants in 
advance of the program. 

Evaluators in this case were able to discover a mosaic of cultural 
images, educational hopes, and communicative dispositions among partici- 
pants by generating information in three ways: through questionnaires 
which attended to the condition of participants’ cultural knowledge, 
sensitivity, professional hopes, and instructional agendas; through in-depth 
interviews which revealed participants’ interpretations of program content, 
goals, expectations, an .! levels of cultural sophistication; and through 
explorations of instructional materials and/or staff development programs 
that had been developed in advance of the program. 

In addition to eliciting basic demographic, instructional, cultural, 
and educational information about participants, the evaluators of the 
National Intercultural Education Leadership Institute generated a ques- 
tionnaire (1990, 1991, 1992, 1993) that revealed participant images of the 
target culture; their knowledge of scholarly literatures, instructional mate- 
rials, and community resources; and their levels of cultural experience, 
association, and communication with informants from other cultures 
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generally and the target culture specifically. The following questionnaire 
.provided an evaluative baseline aiad a guide- focprograin direction: 

Please answer each of the following questions in 250 words or less: 

■ What cultural problems/issucs are most important to you as a 
cultural educator. In light of these problems/issues, what cultural 
content and dimensions do you wish to strengthen? 

■ What scholarly information, popular books, instructional materi- 
als, or television programs, if any, have been most meaningful for 
your understanding ofjapanese culture? 

■ What kinds of communication with Japanese nationals have you 
had? (e.g., conversations, correspondence, social gatherings, study 
or travel in Japan, homestays, etc.) Please explain. 

■ How can this seminar strengthen /ou as a professional cultural 
educator? 

■ What long-term cultural education initiatives do you hope or 
intend to develop through this seminar? Please be specific, for 
example, preparation of new staff development programs; creation 
of a new unit on Japanese culture for the third grade; preparation 
of sister city programs; building of tcacher-teacher, student- 
student exchanges; transformation of global studies curriculum in 
high schools; etc. 

■ How do you plan to share and disseminate what you learn in this 
seminar? Please be specific, (e.g., new course content, assessment 
measures, focus for staff development, policy reformulation, etc.) 

■ Please describe your levels of cultural and language proficiency. 

Beyond providing the evaluator with invaluable information about 
participant goals, this form of questionnaire lays a foundation for in-depth 
interviews with each program participant in advance of the program. 



n < 
1 } 



* 



33 



PART [: Designing, Defining, tint! Evaluating Cultural Education Programs: The Input Side 



In-depth Interviews 

The importance of sensitively conducted interviews at this initial 
stage of program development cannot be exaggerated. Confidential inter- 
views with participants provide a vehicle for them to reveal a full spectrum 
of hopes, dreams, cultural assumptions, and images. An in-depth inter- 
view, of an hour or two in length, provides a safe communication setting 
within which participants may feel free to reveal the particularities of their 
interest in the target culture, the kinds of cultural resources and informa- 
tion they hope to acquire, and the cultural assumptions that they make 
when they select instructional materials and organize learning. Deep 
interviews enable an assessor to discover what perspectives a participant 
holds regarding the target culture; what resources have shaped the. 
participant's images; what definitions he or she has of cultural education; 
and what hopes he or she holds for continuing education and the educa- 
tion of students, whether children, youth, or adults. Most importantly, 
deep interviews provide opportunities for participants to reflect on their 
experiences with cultural others, anticipate cultural discomforts, and 
identify normally unacknowledged fears or negative feelings. 

The following excerpt, taken from a program planning document 
prepared in advance of a seminar focusing on diversity policies and prac- 
tices in japan, provides an example of the utility of deep interviews as 
sources of detailed information about participants. This “Group Profile” is 
a distillation and synthesis of information gathered by a program leader 
from dozens of hours of interviews with Research Associates of the Na- 
tional Intcrcultural Education Leadership Institute. 

The Research Associates of the third National Intercultural Education 
Leadership Institute lead a variety ofjapan-rclated instructional initiatives 
in the United States. Some are directors and/or specialists who act as 
mediators between university-based programs and precollege schools 
designing curricula, teacher education workshops, staff development 
initiatives, summer seminars, and field study programs. Others are 
leaders of internationalization programs at state and school district levels 
organizing staff development programs that engage teachers, school 
administrators, and others in Japan-related curriculum development 
initiatives, and in the preparation of innovative instructional materials. 
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Still others have had many years of experience in Japan working as cross- 
cultural educators and have returned to the United States to become 
program leaders in University departments, integrating content about 
other cultures into teacher education programs, developing comparative 
education courses, cultivating scholarly exchanges, and otherwise leading 
efforts to broaden the perspective taken within colleges of education and 
universities* Some prepare materials about the United States for the use 
ofjapanese nationals visiting, working, or teaching in the U.S. Ail hope, 
tnrough their work with NIELI, to identify new connections and deepen 
the quality and depth of existing exchanges with Japan. When taken 
together, they constitute a leadership group for collegiate and 
precollegiate, Japan-related instruction serving not only as “gatekeepers” 
for Japan-related instruction, but as organizers of intercultural and 
international programs for colleges, universities, and precollege schools 
as well. 

Dr. P. University professor, teacher educator and author of a book 
focusing on internationalization initiatives in higher education; looks on 
the study ofjapanese culture and education as a way to help teachers 
reflect on the assumptions they hold about other cultures and countries 
and to learn new ways to investigate other cultures. Beyond her efforts to 
educate others, she hopes, herself, to learn about internationalization 
initiatives in Japanese schools, universities, and government agencies. As 
a NIELI Research Associate . . . Dr. P. hopes to meet Japanese faculty 
colleagues who are developing innovative approaches to teaching, 
educational exchange, and teacher education. She has a special interest in 
research centers that study foreign students, establish accreditation 
policies, and otherwise contribute to cultural and educational exchange. 

Ms. A. A curriculum specialist with responsibility for curriculum and 
staff development in social studies, economics, and international studies; 
has designed a project called Direct Exchange, intended ultimately to 
cultivate networks of collaborating Japanese and U.S. teachers. As a 
NIELI Research Associate, Ms. A hopes to meet elementary and second- 
ary school teachers who are currently members of study groups focusing 
on internationalization efforts and intercultural curriculum development 
and, like her, hope to make new and exciting instructional materials and 
possibilities available to teachers. Since Japan serves as a focus for one of 
her most professionally enriching and personally rewarding experiences, 
she is eager to return to learn more, teach more, and otherwise enhance 
the depth and breadth of her knowledge of the people ofjapan. 

Mr. H. A teacher ofjapanese language and of social studies; has visited 
Japan many times during a twenty- two year teaching career. As a NIELI 
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Research Associate, Mr. H. hopes to investigate the portrayal ofjews in 
Japanese Manga (comics) and in increasingly popular “business novels.” 

He-would liketo determine the impact, if any, of such popular writing : 

on Japanese youth and to discover what efforts are being made within 
Japan to combat the effects of minority stereotyping in Japanese literature 
and compare them with efforts in the United States. 

Dr. T. Lead English Resource teacher and Chairperson of a large 
language arts department in a public high school. As an instructional 
mentor and department chairperson working with sixteen teachers, four 
composition assistants, and one instructional assistant, she attempts to 
identify creative instructional materials, innovative forms of teaching, and 
new ways for teachers from several disciplines to work together. As a 
NIELI Research Associate, she hopes to explore educational opportuni- 
ties for young women who do not go on to college and is particularly 
interested in the aspirations and educational experiences of minority 
children and the opportunities that society presents to them. She hopes 
to deepen relationships with counterpart teachers and identify inacces- 
sible and important instructional materials. (International Center for the 
Study of Education Policy and Human Values, 1993) 

When an evaluator consults these kinds of documents as well as 
transcripts, field notes, and/or internal memos, he or she can determine 
how sensitively a program leader has integrated participant perspectives 
into the design of the program. An evaluator might determine the fit 
between the two by comparing participant perspectives and program 
emphases, analyzing the level of detail about each participant, or inter- 
viewing one or two participants about the design of the program. 

If such materials are not available, then it is important for an evalua- 
tor to help program leaders construct interview procedures that make this 
kind of information-gathering possible. By helping program designers 
learn how to engage participants in thoughtful dialogue about the dilem- 
mas of constructing and communicating program goals, reconciling 
diverse interests, constituting content emphases, cultivating appropriate 
communication forms, and creating an effective learning group, an evalua- 
tor can serve in a mentoring role as well as in an evaluative capacity. The 
following interview format has served as a useful guide for program leaders 
and, for that matter, for evaluators who choose to engage in interviews 
themselves: 
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Defining the Task 

(Three to five minutes) . - - 

■ Emphasize the concept of perspective and the hope that, through the 
interview, it will be possible to learn something about the hopes, 
dreams, and professional and personal interests of each participant and 
to integrate their perspectives into the planning and fine-tuning of the 
program. 

■ Emphasize the confidentiality of the conversation. Assure participants 
that you are not trying to judge them, but to understand how they 
think about culture, education, etc. 

E Share the dilemmas of integrating individual interest into a group 
profile that enables planners in the target culture to understand group 
needs. 

Discovering Perspectives 

(30 minutes to an hour and a half or more) 

Pose the question: “When you think about the possibilities of this 
program, what do you hope to learn, to do, to experience, to feel, to 
teach?” 

Thereafter disappear, metaphorically speaking, as much as possible 
thereby giving participants room to imagine and inviting them to speak 
discursively and in their own voices. Intervene only to clarify points they 
are making or to assure that what they are saying is what you take it to 
mean. 

Although some participants will, at first, be hesitant, they will become 
forthcoming if you can improvise new approaches to the conversation, 
build from what they are confident about, find less threatening ways to 
approach the task, help them understand how important each particular 
perspective is to the creation of a communicative group profile, and 
muster sufficient patience and humor to display a zealous, but non- 
judgmental intention to understand their perspective. 



Instructional Materials 

A final way to explore the cultural repertoires of participants in 
advance of a program is to observe their classes and analyze samples of their 
instructional materials, exploring the implicit definitions of culture they 
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employ, their knowledge of resources, their sensitivity to the target cul- 
ture, and their relative attention to cultural comparison, cross-cultural 
themes, and the value of cultural reflection. 



Mediated Learning: The Knowledge 
Dimension Explored 

A second element in the evaluation of cultural education programs 
involves the analysis of cultural learning as it is informed by the quality of 
information being made available to A articipants. Evaluators need to 
identify the culture-revealing features of programs in two ways: (a) by 
analyzing the content of instructional materials, and (b) by evaluating the 
evolving capacity of participants to identify and reflect upon the points of 
view espoused by required readings, viewings, lectures, and other instruc- 
tional materials and resources. 

Mapping Content 

Content analysis provides an ideal way for evaluators to identify 
subject matter, perspectives, and thematic elements in instructional mate- 
rials, whether the materials are publications, lessons, statements of goals 
and objectives, syllabi, schedules of seminars, correspondence, or interview 
transcripts. Quantitatively-based content analysis can reveal the subject- 
matter focus, the number and kind of thematic elements, and the array of 
cultural perspectives made available in a program. Beyond the classification 
of content and themes, content analysis also allows for the assessment of 
image-making qualities of instructional materials. Qualitatively-based 
content analysis centers on the exploration of written materials as they 
convey images of culture, reveal matters of perspective as well as matters of 
fact, reinforce stereotypical images or transcend them, and/or stimulate 
exchanges of resources and ideas or disable them. Here, the outcomes are 
expressed not as number values but as relative degrees of cultural sophisti- 
cation, ranging from a simple matter of fact on one end, to complex 
political or ideological stereotypes or characterizations on the other. An 
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image-centered form of content evaluation can reveal ways in which a 
program may have reinforced, transformed, and/or moderated the sub- 
stance of cultural understanding and knowledge for program participants 
(Finkelstein, 1996). 

Monitoring Teaching and Learning 

Another way for evaluators to assess the culture-revealing feature p of 
a program is to gather information from leaders and participants on a 
systematic and ongoing basis. One useful form of evaluation involves the 
preparation of questionnaires that call on leaders and participants to char- 
acterize each instructional material at the time of first reading by attending 
briefly to the perspective of the author, the focus of the material, and the 
features of the culture that were revealed and/or obscured. The following 
“Daily Feedback Instrument,’’ used in a Fulbright Group Study Abroad 
Program in 1993, provides one model. This instrument invited partici- 
pants to reflect upon what they had seen and, by doing so, revealed their 
evolving cultural understandings: 

Daily Feedback Instrument 

Please comment briefly on each of the following questions: 

1. What was the focus of the reading? 

2. What was the author’s perspective? 

3. What did today’s reading reveal about the culture we are studying? 

4. What did today’s reading conceal about the culture? 

5. In what ways, if any, did the reading reinforce or challenge your image 
of the culture, what you already know, and/or some previously taken- 
for-granted assumption? 

6. What utility, if any, did today’s reading have on the lessons you might 
plan, the professional workshops you might organize, the way you 
assess student learning, or think about the target culture? 

In addition to the use of a questionnaire, a second kind of evaluation 
becomes possible when a participant evaluator is attached to a group on a 
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daily basis. When this happens, the evaluator can gather information by 
engaging in informal conversation, keeping notes, or generating transcripts 
that reveal the evolving reflections of participants as they experience each 
feature of the program. Not uncommonly, group discussions and travel 
between places serve as particularly fruitful occasions for an evaluator to 
explore perceptions. The following notes were kept by an evaluator in 
1993 who wanted to understand why participants were particularly sur- 
prised by a video that documented what participants believed to have been 
certain disorderly qualities in Japanese preschools: 

It appears that participants had thoroughly integrated visions of Japanese 
education as uniform, orderly, almost military in the quality of discipline. 

In fact, three of the participants were shocked by what appeared to them 
to be a certain indifference to displays of physical violence or verbal 
abuse, having not yet learned that delegating responsibility to the student 
group was a feature of group-building in Japanese schools. The film, 

Preschool in Three Cultures , seems to have had an uncommonly deep effect 
on participant images of Japanese schooling. As an instructional material, 
it helped many transcend stereotypes. 

A participant evaluator can also observe and analyze the quality of 
reflective conversation that evolves during systematic discussions of cul- 
tural information. When evaluators take these kinds of opportunities, they 
are strategically positioned to identify the intellectual and cultural power of 
particular information sources and to attend to the teaching value of each 
informant’s approach to cultural information. Discussions of instructional 
materials, of the challenges of learning certain materials, and of the rigors 
of learning new habits of heart and mind constitute opportunities for an 
evaluator to follow the evolving capacities of participants to bring new 
perspectives, sources of information, and reflective dispositions to what 
they have been learning. 

In the case of the Fulbright Group Study Abroad program, the 
responses of participants revealed transformations in their cultural con- 
sciousness: “I can’t believe that Japanese teachers are so tolerant ” “This 
film completely transformed my view of Japanese education.” “I am going 
to have to rethink my approach to teaching about other cultures” (1993). 
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Learning to Observe and Reflect in 
Cultural Encounters: Detecting 
Communicative Dispositions 

Yet another way to identify the power of a program is to assess both 
the availability and quality of opportunities designed to help participants 
remain centered and reflective when they encounter situations that im- 
merse them totally in settings with cultural c’ hers. An evaluator who is 
aware of the possibilities of ethnography ana field observation as cultur- 
ally-enabling reflective techniques can call upon program leaders and 
participants to keep and share reflective journals, track improvisation 
strategies systematically, and otherwise prepare information that will create 
opportunities for reflection and data collection. Indeed, evaluators have an 
important role to play as cultural diagnosticians who, as they review 
participant responses to culturally strange settings, can both advise pro- 
gram leaders and assess participant learning. One especially useful approach 
to this strategy was demonstrated by a program which was designed, in 
part, to help participants integrate cultural exchanges into instructional 
units in professional development, social studies, global learning, and other 
culture-laden subject matters. The program engaged participants with 
experienced ethnographers who had cultivated an array of techniques to 
help participants appreciate the usefulness of “ruptures of communica- 
tion” — moments when understanding ceases, confusion abounds, and the 
need for sense-making acquires new dimension (Agar, 1993). In response 
to a written question about the value of these kinds of sessions, participants 
articulated an important feature of ethnographic preparation. As one 
wrote, “[It] is very useful to help us refrain . . . from immediately inter- 
preting new encounters using existing frames of reference.” Another 
suggested that the session honed her ability to “reframe how I could talk tc 
people in a new way to get the info I’m looking for.” 

As a result of the techniques advocated by another ethnographer, 
who emphasized the value of keeping field notes and reflective journals, 
participants took opportunities to practice “using themselves as research 
instruments,” “suspending judgment to discover perspective,” “bracketing 
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their values,” preventing over-identification, and resisting the temptation 
to makejthe familiar strange or the strange familiar, or to generalize from a 
single experience (Shimahara, 1992). 

Cultural Encounters 

Beyond exploring the impact of scholarly information, cultural 
images, and other mediated materials, evaluators also must attend to 
another feature of cultural learning: encounters with cultural others. This 
feature of cultural teaching and learning has traditionally occurred in 
programs with study abroad components and not typically in one-day, 
one-week, or even one-month cultural studies programs. Cultural en- 
counters seek to provide participants with new forms of experience to 
reflect upon what they have learned; to revisit the limitations ,.nd possi- 
bilities of mediated information; to experience alternative habits of cultural 
knowing, doing, feeling, and believing; to hone their communication 
skills; and to revisit their own cultural habits. 

In our view, one of the most important questions an evaluator can 
ask about any cultural education program is whether it integrates opportu- 
nities for participants to cross cultural borders and experience life in what 
Min-Zhan Liu has called cultural contact zones (Eddy, 1996). For an 
evaluator, it is important to assess the role of these cultural encounters in a 
program, describing how such encounters reveal and/or obscure perspec- 
tives from within target cultures; how they prepare participants to observe, 
describe, and analyze cultural events; and how they train participants to 
anticipate cultural dissonances, learn directly from cultural informants, and 
rebuild educational communities. A culturally astute evaluator can gather 
information systematically by attending to three culturally evocative 
possibilities: the power to discover new perspectives; the power to observe 
habits of heart, mind, and association in alternative culture settings; and 
the power to reflect upon and learn from the previously unfamiliar. If, for 
any reason, the company of cultural others and opportunities for immer- 
sion in unfamiliar settings is inaccessible, the experience of a cultural 
encounter can be simulated through object, film, and photograph and in 
restaurants, arboretums, or markets. 
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Perspective-taking 

Since no single encounter, no matter how long in duration or inten- 
sity, can illustrate the complexity of a particular culture, it is important for 
an evaluator to take account of the perspective-developing possibilities of 
cultural encounters as well as those of mediated materials. and group 
reflections. An evaluator needs to generate an understanding of the per- 
spective-revealing qualities being made available to participants and gener- 
ate an assessment of the power of these encounters to induce participants 
to transcend cultural stereotypes. 

In cultural education programs of short duration, an evaluator would 
do well to analyze the relative availability of cultural informants and en- 
counters. On college and university campuses, potential informants might 
be visiting scholars or foreign students. Outside the collegiate setting, 
experts on business and trade can be located through local chamber of 
commerce offices and city-sponsored international trade councils. Artists, 
writers, or politicians from the target culture might be found among 
people who work at museums, local libraries, embassies, or consulates. 
Cultural societies, such as the Japan-America Society, the Hellenic Neigh- 
borhood Association, or the Initiative for Peace and Cooperation in the 
Middle East, are useful sources for cultural informants. Cultural experts 
with deep experience in the target culture and with education experience 
are sometimes helpful. 

If, for example, the program is a one-day, in-service workshop on 
Japanese education, and the cultural encounter is a discussion with a 
Japanese exchange teacher from an elementary school in Tokyo, the 
evaluator would want to see what opportunities participants had been 
granted to explore alternative viewpoints, using works as varied as Thomas 
Rohlen’s Japan's High Schools , Merry White’s The Japanese Educational 
Challenge , videos such as Tobin, Wu, and Davidson’s Preschool in Three 
Cultures , and films such as Twenty-four Eyes directed by Keisuke Kinoshira. 

If a program were to engage participants in longer-term explorations 
of culture as it might be revealed in academic courses, then an evaluator 
would need to discover the range, multiplicity, and complexity of view- 
points provided for participants. If an evaluator were to assess a program in 
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which participants had the opportunity to read an array of government 
documents and approved textbooks; to speak.with government officials; to 
meet with local, prefectural, or national superintendents of schools; and to 
converse with publicly-appointed curriculum supervisors, then he or she 
would note that participants had been given a rich though partial view of a 
culture — an official, policy-driven one. This kind of encounter, if uncor- 
rected or unreflected upon, might reinforce a stereotype that education in 
Japan is relentlessly centralized, serves a homogeneous population, and 
varies inconsequentially from community to community — a common, 
but, in fact, incorrect view. 

If, however, participants were to visit an elite academic high school, 
talk to students in a private “cram” school about their preparation for 
college entrance exams, and hear a lecture by a college professor on the 
trials of “examination hell,” they would acquire yet another interesting but 
partial view on the culture. Since these encounters engage participants 
with motivated teachers and students in advanced classes in expensive cram 
schools, the experience could reinforce a stereotype that all students in 
Japan spend their adolescence in an educational rat race to enter a presti- 
gious university. In fact, less than 50% of the students do. If, as is so fre- 
quently the case, encounters reveal only bits and pieces of a culture and 
provide no countervailing points of view, then stereotyping will almost 
certainly occur — a situation which an evaluator must take care to note. 

Beyond peeling apart the culture-revealing features of each encoun- 
ter, evaluators must also assess the educational power of an encounter by 
constructing questions that determine participants’ evolving abilities to 
discover previously unnoticed or invisible perspectives. The following 
questions, designed for a 1994 curriculum transformation initiative that 
linkedjapan study to instructional improvement in world culture courses, 
called on participants to assess each teacher encounter by keeping journals, 
diaries, or narratives that addressed issues of perspective in substantive ways: 

■ What, if anything, did this encounter add to your knowledge of 
Japanese culture and society? What perspectives were revealed? 
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■ What, if anything, did the encounter reveal about Japanese pat- 
terns of communication and the uses that Japanese counterparts 
would make of E-mail communication? 

S How, if in any way, did it add to your ability to communicate with 
Japanese counterparts? 

■ What similarities and differences did you notice in the way coun- 
terparts thought about the uses* of E-mail exchange in classrooms? 

Each of these questions called on participants to consider the cultural 
learning possibilities of each encounter as well as to reveal their evolving 
understanding of insider perspectives. In this way, the evaluator was able to 
link program content to cultural learning as participants actually acquired it. 

Reflection 

It is possible to generate information by asking participants to prepare 
summaries and/or recordings of debriefings, to keep reflective journals, to 
create pictorial representations of their encounters, to share poetic evoca- 
tions, etc. Beyond analyzing these products for what they reveal about the 
evolving cultural sophistication of participants, an evaluator can, while the 
program proceeds, also institute the staple of cultural education evalua- 
tion — the deep interview conducted through private conversation, fac- 
simile, or E-mail. If this is impossible, then evaluators will need to develop 
strategies to gather information retrospectively through questionnaire, 
analysis, and interview. 

The follow-up to a cultural encounter, the opportunity to reflect 
upon the encounter and to share reflections, is an important element in 
the continuing learning of participants- — as important, in our experience, 
as the encounter itself. Good program leaders and evaluators encourage 
participants to consider what they learned from the encounter, both 
through self-reflection and by sharing their thoughts. Cultural debriefings 
are as indispensable in short-term one-day workshops as they are in more 
sustained programs with study abroad elements. After a particularly de- 
manding or stressful cultural encounter, participants are often bursting 
with questions and comments: “Do you think I offended him with my 
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question?” “She wasn’t at all the demure Japanese woman I expected.” 
“What did you make of her reaction to our gift?” “Did our speeches __ 
communicate well?” “When I hung my coat in Professor H’s shrine do 
you think he was offended forever?” If an evaluator conducts the debrief- 
ing, then he or she would generate data that reveals the thinking processes 
and intercultural skills of participants. If not, the assessment of this element 
would require an analysis of leader debriefing strategies and participants’ 
journals or other written reflections. 

Not only are participants generally eager to explore their reactions 
and exchange perspectives on situations that confuse and confound, they 
are also able, through the medium of debriefings, to revisit their own 
perspectives as well. It is important that participants have opportunities to 
air their views in educationally safe havens where rights to privacy are 
protected, honesty is valued, a non-judgmental atmosphere prevails, and 
humor abounds. Our debriefings over the years have revealed delightful 
cultural mistake stories through which program leaders and participants 
alike reveal interpretive errors, acknowledge cultural faux pas, and discover 
perspective mistakes. This forms the foundation for new levels of under- 
standing. The following story (1993) is illustrative: 

An American teacher was observing in an elementary school in Japan 
when she became fascinated with the displays of the children’s artwork in 
the school’s halls. She had studied child psychology and noticed that 
many of the Japanese students had drawn pictures of children in great 
pain. The children in the drawings had their mouths wide open, as if 
screaming or crying, and tears streamed from their eyes. The American 
teacher thought, “These Japanese children look happy and carefree but, 
judging from their drawings, inside they must be seething with anger and 
pain.” She wished she could read the Japanese writing on the posters for 
more insight into the inner turmoil of the children. Then she took a 
closer look at one of the drawings and consulted with a Japanese speaking 
leader. Next to the crying child was a toothbrush and a tooth with a large 
black spot on it. The American teacher then realized her error. The 
poster did not signal inner turmoil. Rather, it revealed the consequences 
of unbrushed teeth and a child’s response to tooth decay. 

Data-generating strategies through cultural debriefings serve both 
educational and evaluative purposes if participants, program leaders, and 
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evaluators have formed a foundation of trust and respect. Indeed, it is 
possible to track transformations in the capacity of participants to integrate 
insider perspectives, discover new sources of information, communicate with 
strangers, and otherwise overwhelm obstacles to learning in strange places. 



Retrospective Cultural Reflection: 

Vision, Knowledge, Communicative 
Disposition, and Community Building 
in the Wake of Programs 

The time for retrospective cultural reflection, when participants 
revisit the whole of their cultural experience and make plans to apply the 
knowledge they have acquired, is an auspicious one for participants and 
evaluators alike. It is a time when participant memories are fresh and their 
experiences as new cultural learners are immediate. It is also a time when 
discrete features of the program stand out, unrcfracted by other concerns. 
This is when participants can begin to fashion ways to apply what they 
have learned to their cultural and educational practices and let their imagi- 
nations fly. 

An alert evaluator will take advantage of this time and generate 
opportunities for participants to identify the features of the program that 
have been most useful for the development of their intellectual, pedagogi- 
cal, instructional, and cultural capacities, as well as their evolution as 
cultural educators. 

Questionnaires 

One way to identify which features of cultural education programs 
their participants find most useful is to modify and re-administer previ- 
ously prepared questionnaires and, through a comparison of results before, 
during, and after a program, assess levels of cultural sophistication as they 
have evolved over time and in response to program opportunities. Ques- 
tionnaires might also be gathered from matched non-participants, thus 
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generating comparative data as well. The re-administration of a retrospec- 
tive Likert-like instrument that calls on participants to assess each program 
element once again provides comparative data that can index any changes 
in participant assessments ofdiffc. ent program features. 

Another form of questionnaire foregrounds program goals and calls 
on participants to identify programs and program features that may have 
helped them cultivate perspective-taking skills, cross-cultural communica- 
tion capacities and dispositions, knowledge of new instructional resources 
and documents, and/or a capacity for cultural reflection. The following 
questionnaire, prepared in 1994 by the International Center for the Study 
of Education Policy and Human Values for a three-year program that 
linked Japan study to the improvement of instruction in inner-city schools, 
illustrates the form: 

■ What program features, for example, readings, encounters, 
debriefings, and opportunities for reflection, revealed approaches 
to cultural problems and issues that are important to you as a 
cultural educator? 

H What features, if any, of this program have strengthened you as a 
professional cultural educator? 

■ What forms of information have you found most/least useful for 
your understanding of culture generally and Japanese culture 
specifically? Please allude to various kinds of information such as 
scholarly articles, books, and information from cultural informants 
(e.g., homestay families, taxi drivers, diplomats, teachers), cultural 
sites, community resources, instructional materials, and television 
programs. 

■ What particular features of the program, if any, led you to modify 
the way you prepare staff development programs, create new 
instructional units, deepen teacher-teacher and student-student 
exchanges, transform global studies curriculum in high schools, etc.? 

^ What new cultural skills, if any, do you think you have acquired as 
a result of this seminar? 
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Calls for reflective essays constitute yet another way for evaluators to 
generate information about the educational power and utility of a pro- 
gram. The following document (1993) invited participants to provide a 
retrospective overview of a program that linked the study of educational 
purposes, practices, and policies in Japan to instructional improvement in 
the U.S. Entitled “Japan from Inside Out,” the program engaged partici- 
pants in sustained eight-week study that included: (a) a graduate course 
comparing Japanese and U.S. educational practices and policies as reflected 
in scholarly literature, popular imagery, Japan-related instructional materi- 
als, and local community resources; and (b) a five-week program of teach- 
ing and learning in Japanese families and schools. At the end of this pro- 
gram, the evaluator created an instrument designed to provide program 
participants with opportunities to assess the effects of the program. 

Japan from Inside Out: A Final Evaluation 

For this evaluation, we would like you to compose a reflective essay on 
your cultural education experience. The information collected through 
these essays will help program planners enhance the quality of future 
programs and will help us to understand what features of the seminar 
have been most valuable for you. This essay should be typed, approxi- 
mately three pages long, and address these issues: 

■ How has this seminar changed your perceptions about Japanese 
culture and the Japanese educational system? 

■ How did this seminar affect your perceptions of your own culture and 
educational system? 

■ Describe how your experiences might change your teaching strategies 
in the future. Provide specific examples of how you might incorpo- 
rate program experiences into your professional life (e.g., instructional 
materials, professional presentations, staff development efforts, etc.). 



Deep Interviews 

A final way to generate information that reveals the relative utility of 
various program features as well as the overall impact of a program is to 
conduct interviews that mirror and parallel those that were conducted in 
advance of the program. Sensitively handled interviews provide opportu- 
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nities for participants to reveal concerns that would otherwise be invisible, 
such as their view of leaders as intellectual, cultural, and civic mentors and 
role models; their discomfort with alternative cultural settings and com- 
munication forms; and their capacities to take advantage of different 
program opportunities, to name a few. 

Interviews also invite participants to identify features of the program 
that they believe will be most useful to them as they integrate what they 
have learned into their professional repertoires. The deep interview can 
reveal specificities and details that would otherwise go unattended. For an 
evaluator, the deep interview constitutes an information source from 
which he or she can assess the relative coherence of the program and 
disaggregate its various features by relating program elements to the dispo- 
sition of program participants to make use of them. 

Taken together, the contents of retrospective questionnaires, essays, 
and interviews enable evaluators, leaders, and participants to assess pro- 
grams as they unfold. Ultimately, however, the power of a program will be 
revealed when an evaluator focuses on the capacity of participants to design 
their own cultural education programs, prepare sophisticated instructional 
materials, cultivate communicative dispositions, create new forms of 
cultural practice in classrooms, and otherwise engage in a variety of cultur- 
ally educative professional activities. These applications constitute both the 
last phase of a program and an opportunity for summative evaluation. 



Cultural Application: 

Community-building Dimensions 

Beyond exploring the impact of scholarly information, cultural 
images, and encounters on the evolving cultural repertoires of participants, 
evaluators might consider one last, often-overlooked educational assess- 
ment possibility in cultural education programs: the discovery of new 
resources for and approaches to teaching about other cultures. In programs 
of study that engage participants in the exploration of previously un- 
known, inaccessible, or “truly foreign” cultures, it is often true that effec- 
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tive models of instructional materials are, relatively speaking, inaccessible 
and/or unavailable. An important question for an evaluator to pose is- 
whether program content limits opportunities for participants to evaluate 
the instructional materials and consider new resources and strategies for 
teaching about cultures generally, and about the target culture specifically. 

Alert evaluators should be on the lookout for program opportunities 
that invite participants to evaluate existing resources for teaching about 
another culture; to critique the quality and utility of what is already avail- 
able; and to reflect on how they might integrate new human, instruc- 
tional, and material resources into their teaching repertoires. Evaluators 
might call on participants to identify resources that they intend to use as 
they prepare, coordinate, and/or conduct cultural education teaching and 
learning programs. 

An interesting form of program evaluation that is seldom undertaken 
is one that attends to community-building and reveals the effects of a 
cultural education program on the evolving professional repertoires of 
program participants. When participants apply what they have learned 
after a program is done to the preparation of other cultural education 
programs, instructional materials, cultural encounters, modes of assess- 
ment, curriculum transformations, etc., it is possible for an evaluator to 
index program quality to behavioral outcomes. 



Bibliography 

Agar, M. (1993). Excerpts from a lecture delivered to the National Intercultural Education 
Leadership Institute. College Park, MD: International Center for the Study of 
Education Policy and Human Values. 

Eddy, R. (Ed.). (1996). Reflections on tnulticulturalistn. Yarmouth, ME: Intercultural Press. 

Finkelstein, B. (1996). Evaluating the content of USJF initiatives: The knowledge dimension. 
In B. Finkelstein (Ed.), Designs for program evaluation: Assessing the work of USJF precollege 
education programs (pp. 1-8). College Park, MD: International Center for the Study of 
Education Policy and Human Values. 

International Center for the Study ofEducation Policy and Human Values, (1990, 1991, 1992, 
1993). National Intercultural Education Leadership Institute group profiles, College Park, MD: 
Author. 

Shimahara, N. (1992). Excerpts from a lecture delivered to the National Intercultural 

Education Leadership Institute. College Park, MD: International Center for the Study 
ofEducation Policy and Human Values. 




51 




Evaluating Cultural 
Education Leadership 



V 



r j 
Ob 




PART I: Designing, Defining, and Evaluating Cultural Education Programs : The Input Side 



■ The primary purpose of evaluating cultural 
education leaders is to assess and improve their 
performance and, by extension, their programs. We 
hope in this chapter to help evaluators identify and analyze the features of 
education programs over which leaders exercise choice and judgment. We 
provide a template with which to evaluate the quality of cultural education 
leadership by describing what cultural education leaders do and how they 
do it. Evaluators must be prepared to be surprised and even delighted by 
what they find — and ready to rethink their own assumptions about leaders, 
evaluation methodology, and the study of culture. 

Our definition of leadership includes the deliberate shaping of events 
and the purposeful facilitation of change. A core assumption of this chapter 
and, indeed, this book is that educators who consciously involve the study 
of culture in a formal academic curriculum are leaders. These educators 
hold positions and engage in actions that potentially influence the thinking 
and behavior of others. “Others” in this context may be students, col- 
leagues, administrators, curriculum reformers, school improvement activ- 
ists, community change agents, parents, governing and advisory board 
members, philanthropists and other funders, the media, the general public, 
and so on. 

Leadership evaluation often takes place in classrooms, where teachers 
develop curricula and where learners are invited to think in different ways 
about culture. Other settings include study abroad programs for secondary 
and higher education students, staff development programs for educators, 
and cultural education seminars for business persons and community 
leaders. Consequently, leaders, as discussed in this chapter, may be teach- 
ers, administrators, education policy makers, professors, international 
studies program coordinators, curriculum specialists, corporate consultants 
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and trainers, or program directors. The programs under evaluation may 
include curriculum development projects; staff development initiatives, 
including pre- and in-service activities; and international study excursions. 

As we have noted in previous chapters, we believe that there are 
four characteristics of cultural education programs that warrant the atten- 
tion of evaluators: leaders’ vision, knowledge, communicative disposition, 
and capacity for community building. Taken together, these characteristics 
provide a wide-angle lens through which the actions and words of cultural 
leaders can be studied and judged. In the following four sections of this 
chapter, we discuss each of these characteristics in turn, posing several 
evaluation questions, suggesting places where evaluators might gather 
information to answer the questions, and offering examples from actual 
evaluations to illustrate our points. 



Leader’s Vision 

Education leaders, like all humans, import culture into their own 
work. As a result, leaders’ concepts and beliefs about culture and cultural 
education, approaches to diversity, and cultural expectations encoded in 
statements of program goals and objectives — and the extent to which these 
influence leaders’ professional practices and world views — are of primary 
importance to program evaluators. In this section on program leaders’ 
vision, we discuss the importance of having a clear vision of cultural 
education, some of the common visions cited by cultural education leaders 
today, places where the program evaluator can hunt for evidence of lead- 
ers’ vision, and the problem of discrepancies between espoused vision and 
actual practice. 

A Vision of Cultural Education 

A well-developed personal vision of cultural education is important 
because it can give a program leader a moral compass with which to navi- 
gate the waters of competing and conflicting professional demands. It is 
important for determining and then staying on course, and also for decid- 
ing what is important. Visions overcome resistance to change (Bennis & 
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Nanus, 1986) and can be used to paint a future that people can identify 
with and get excited about. Personal visions usually reflect deeply held 
beliefs, principles, and values (Apps, 1994). The personal vision of the 
cultural education program leader can fundamentally affect program 
design and program outcomes. 

Many cultural education leaders lack a well-thought-out vision of 
cultural education. Others may have one but find it hard to articulate. If a 
leader has difficulty articulating his or her vision, then there is a risk that 
the program elements will not reflect a coherent philosophy or will consist 
of half-baked ideas and desires. Consequently, by focusing the spotlight on 
leaders’ personal visions, the program evaluator can encourage leaders and 
others interested in the success of the cultural education program to reflect 
on what they are doing and why, and to consider a range of alternative 
perspectives. 

Is one vision better than another? Yes. In general, we believe that a 
more sophisticated vision — one that might, for example, emphasize the 
development of multiple perspectives from within a culture — is superior to 
a vision that simply emphasizes the acquisition of factual knowledge about 
a single culture. We also believe in the importance of creating educational 
spaces where people can observe how different cultures come into contact 
with one another and the mutual influence that this contact often pro- 
duces. 

It is important to recognize that an evaluator is not simply a cheer- 
leader for one or another vision. Instead, it is useful for the evaluator to be 
aware of a range of possible visions for cultural education leaders, for • 
opportunities will arise in which the evaluator can suggest alternative 
directions in which the program and leader might have gone. Obviously, if 
a leader is unaware of alternative visions of cultural education, this might 
suggest that the he or she is also unaware of knowledge and practices that 
might improve program content and outcomes. 

Three Common Visions of Cultural Education 

The following three examples of visions represent fairly common 
beliefs concerning the possibilities of cultural education as cited by today’s 
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leaders. They are: (a) to become more multiculturally aware within one’s 
own community and nation, (b) to extend one’s understanding of culture 
beyond one’s national borders, and (c) to develop skills and strengthen 
dispositions that support democratic principles and institutions. 

Becoming Multiculturally Aware: One vision involves a belief in 
the educational possibilities of cultural diversity and the need to acquire 
certain sensibilities and skills in order to live successfully and benevolently 
in a multicultural community and world. This may be similar to what 
Mary Catherine Bateson (1994) calls the importance of developing “pe- 
ripheral vision.” She argues, "Because we live in a world of diversity, we 
are privileged to enter, if only peripherally, into a diversity of visions, and 
beyond that to include them in a range of responsible caring” (p. 12). 
Anthropologist Clifford Geertz puts it another way: “We’re going to be in 
each others’ faces more, and so we have to confront the seemingly irrecon- 
cilable differences between ‘Us’ and ‘Them,’ a task hard to do now, when 
the emphasis in the socie^ at large often seems to be on a quest for hard 
certainties” (Berreby, 1994, p. 44; see also Geertz, 1995). 

Going Beyond Your Borders: Other leaders see in cultural educa- 
tion the chance to help learners understand and profit from diversity, not 
only in their own midst but also beyond their immediate surroundings to a 
larger world in which individual Americans and the nation as a whole have 
both interests and obligations. These leaders see potential danger in the 
tendency today for some Americans to withdraw into their own borders, 
both territorial and psychological, in the belief that people in or from other 
countries are responsible for such U.S. problems as large trade deficits, 
unemployment, or government payments to immigrants. These leaders 
believe a reversal of this trend can be effected in part by helping people 
make the connection between private troubles — including ones that 
haven’t been named — and broader political processes. 

Culture and Democracy: Other leaders see in cultural education 
the possibility of strengthening the foundations of American democracy. 
Worried that the ties that bind people from different backgrounds are 
fraying in this country, these leaders feel an obligation to promote what 
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Bethke (1995) calls “democratic dispositions,” a set of attitudes that in- 
volves “preparedness to work with others different from oneself toward - ... 
shared ends” (p. 2). This working with “others” requires learning new 
attitudes and skills, including the ability to listen, show respect for people 
with different views, discuss disagreements thoughtfully, and make com- 
promises. The need to understand other cultures becomes for these cul- 
tural education leaders an essential element of democratic education in a 
multicultural society. 

Hunting for Evidence of Vision 

In the hunt for evidence of a leader’s vision, it is important for the 
evaluator to interview the leader in depth. Equally important, however, is 
the examination of a formal mission statement or other program docu- 
ments that explain succinctly what the program is trying to accomplish and 
why. This information might be found in grant proposals, application 
forms, missives to participants, or correspondence written by the leader to 
funders, supporters, or cultural informants. It is helpful if the program 
leader’s personal vision is in harmony with, or at least not in opposition to, 
the program vision, for from the program’s vision flows the goals and 
objectives. 

If the leader’s vision for the program cannot be found in program 
documents, it may suggest one or more of the following: a possible dis- 
crepancy between personal beliefs and program goals; outcomes that will 
be other than what the leader expects; muddy, narrow, or incomplete 
thinking that will have an adverse affect on achieving program goals; or 
lack of pedagogical know-how. 

Contradictions Between Espoused Vision and Actual Practice 

It is important for evaluators to be especially mindful of discrepan- 
cies, inconsistencies, paradoxes, contradictions, and omissions or “struc- 
tured silences and absences” (Tripp, 1993) between the leader’s espoused 
personal vision and how the program is designed and implemented. For 
example, if a program leader says he or she believes strongly in cultural 
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diversity and then demonstrates the opposite in his or her selection of 
participants for a cultural education program, such a finding could be very 
important to the overall program evaluation. Similarly, if the leader es- 
pouses the importance of creating a democratic climate in the program by 
encouraging discussion, analysis, and debate, but then proceeds to quash 
any hint of controversy or criticism, this contradiction would constitute an 
important finding upon which recommendations for improving program 
management could be based. 

Our own experience with evaluating a cultural education seminar for 
teachers and administrators revealed a similar discrepancy between vision 
and practice. According to the program director, an important part of the 
seminar was to develop cultural leadership skills in the participants, all 
selected because of their presumed experience as education leaders or 
potential leaders. Yet when examining the program documents and other 
data sources, there was little evidence that the subject of leadership in a 
cultural context had been addressed in the curriculum. Apparently, the 
director had assumed that because the participants were already education 
leaders, there was no compelling need to construct formal discussions or 
sessions around the topic of cultural leadership. As a result, opportunities 
in this program for shared learning experiences may have been lost. 

Once the discrepancy was pointed out, the program director modi- 
fied the content of the curriculum to include presentations and discussions 
on the topic of cultural education leadership. As it turned out, this feature 
of the program, especially the discussion among the participants, was 
highly rated by the participants. When utilized by the program designers, 
the knowledge and experience of participants became an important source 
of collaborative learning. 



Knowledge 

A second feature of cultural education leadership that warrants the 
attention of evaluators is knowledge — both the leader’s own knowledge 
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and ability to conduct the program, and the kind and quality of knowledge 
that program leaders make available to participants. In this section we pose 
the following seven questions to frame evaluation issues about knowledge: 

■ Does the leader have the knowledge and experience to lead the 
program? , 

■ Does the leader know where to look for expert cultural assistance 
and advice? 

■ What kind of knowledge does the program leader make available 
to participants? 

■ Does the leader reveal an understanding of the value of multiple 
perspectives in the study of another culture? 

m Does the leader emphasize problems and dilemmas as a way to 
develop participants' knowledge about their own and other cul- 
tures? 

■ Does the leader provide opportunities for participants to reflect 
upon themselves as bearers and creators of culture? 

■ Does the leader help participants apply cultural learning in their 
workplace settings? 

Does the leader have the knowledge and experience to lead 
the program? 

One question that evaluators can pursue in the knowledge dimen- 
sion is whether the leader is competent and capable ofleading a cultural 
education program. Much has been written about the qualities of good 
leaders in general (Bass, 1981; Bensimon, Neumann, &Birnbaum, 1989; 
etc.). Little is available, however, on the qualities of leaders in the field of 
cultural education. 

In our view, in addition to the qualities that make a good leader — 
excellent communication skills, a vision that inspires, superior organiza- 
tional ability, etc. — the ideal cultural education leader will be knowledge- 
able about the field of education, familiar with the dilemmas of cultural 



Chapter 3: Evaluating Cultural Education Leadership 



teaching and learning, and sensitive to the needs of teachers. Ideally, he or 
she will be aware of the pitfalls of overgeneralization and stereotyping, of 
oversimplification and instant judgments in the absence of evidence, and 
of unexamined assumptions about culture. The model cultural education 
leader will be observant, self-reflective, comfortable with ambiguity, and 
open to alternative interpretations of events. He or she will be able to 
improvise in new cultural situations and, perhaps most important, will 
maintain a sense of humor in the face of cultural challenges. 

Evaluators have many avenues to pursue for assessing the ability of 
the leader to conduct a cultural education program. First, they can rely on 
their own judgment as they observe the leader at work and interview him 
or her in depth. Second, they can question the participants who, in our 
experience, are as willing to praise a good leader as they are to criticize a 
poor one. Third, they can talk with the cultural informants with whom 
the leader works to create and implement the program. Fourth, they can 
examine the leader’s background for previous experience in another 
culture or for knowledge of cultural teaching and learning. The leader’s 
background might include: formal course work, foreign language ability, 
experience living abroad, travel, talks or presentations about the culture, 
participation in national and international professional networks, and 
published writings such as professional journal articles, curriculum materi- 
als, book reviews, and letters to editors. Evaluators can also look for evi- 
dence of the leader’s familiarity with the major works and authors dealing 
with the target culture and with knowledge of specific curriculum re- 
sources suitable for participants and their educational settings. 

Does the leader know where to look for expert cultural 
assistance and advice? 

An ideal program leader will be both a professional educator and an 
expert on the culture being studied. In our experience, however, this is 
rarely the case. Sometimes we find a cultural expert who attempts to put 
together a program for educators. Far more common, however, is the 
educator who develops a program focusing on a culture about which he or 
she knows relatively little. Whichever the case, it is important for leaders to 



FART I; Designing, Defining, and Evaluating Cultural Education Programs: The Input Side 



know where to look for expert cultural assistance and advice. In order to 
assess leaders’ abilities to draw on all available resources, the evaluator can- 
ask of leaders: 

■ Are they familiar with their community’s cultural resources, such 
as subject experts and informants from the culture? 

■ Do they know and interact with informants at the local embassy 
or consulate general? How do they make use of these resources? 

■ Do they draw upon expertise at local universities? 

■ Do they make use of civic and cultural associations? 

■ To what extent are they involved in national or international 
organizations promoting the study of nations and cultures? 

■ Are they aware of and do they make use of various national and 
international computer resources, including the Internet? 

■ Do they tap the experiences of alumni from previous programs 
they have directed? 

■ Have they deliberately sought the expertise of other leaders? 

IVhat kind of knowledge does the program leader make 
available to participants? 

Leaders of cultural education programs, like teachers in a classroom, 
function as cultural gatekeepers. They choose what is taught about a 
culture and what is not, which features of the culture come into view and 
which do not, and which perspectives are revealed and which are not. 

They make decisions concerning the type of information worth having, 
cultural encounters worth enabling, and learning occasions worth creating. 
Consider, for example, two programs that engage teachers in the study of 
Japan: In one program, participants learn about kabuki , origami, geisha, and 
the tea ceremony — all fascinating aspects of traditional Japan that also have 
their place in modern Japan. In the other program, the teachers learn 
about culture in Japanese education and Japanese approaches to diversity in 



6 



n 

( 



62 



Chapter 3: Evaluating Cultural Education Leadership 



schools. Given the very different images of Japan that the two groups of 
teachers receive, it is not difficult to imagine the views ofjapan that they 
will pass on to their students. These two examples illustrate the power of 
program leaders to shape, define, and influence knowledge. Evaluators can 
ask leaders directly about their choice of information to make available to 
participants. Even more revealing may be an analysis of the readings, 
videos, objects, and guest speakers used to portray the target culture. 

For an evaluator, an analysis of the program as the leader constructs it 
should expose the following: 

H the power of the content to reveal or conceal important features of 
a culture, illuminate multiple perspectives from within a culture, 
and uncover similarities and differences among and between 
cultures; 

■ the opportunities for cultural observation, reflection, and skill- 
building; and 

■ the instructional^ evocative power of the program — its capacity to 
assist participants to translate cultural knowledge and experience 
into effective instructional materials, new visions of curriculum, 
new frames of cultural reference, and new opportunities for 
professional expression (Finkelstein, 1996). 

There are three issues that an evaluator faces when assessing the 
knowledge dimension of a program. The first is the evaluator’s own 
knowledge about the target culture. Is he or she able to tell when the 
knowledge being disseminated is accurate, sensitive, and balanced? Does 
he or she recognize cultural stereotypes, biases, overgeneralizations, and 
oversimplifications? If the evaluator does not possess the knowledge to 
make these judgments, then it becomes necessary to engage a cultural 
consultant who is able to conduct a content analysis. 

The second issue facing an evaluator is the traditional dilemma of 
disentangling the effects of program content from other influences on 
cultural learning such as the image-making power of mass media; the 
relative influence of early cultural learning in family, community, and 
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school; and the knowledge of the culture with which participants enter the 
program. _For this reason it is e specially important for the evaluator and 
leader to engage in a process of pre-program cultural probing to determine 
participants’ levels of cultural awareness before the program can have any 
effect (see Chapter 2: Evaluating Cultural Education Programs). 

The third challenge for the evaluator is to determine the relative 
impact of one or another feature of the program on an array of potential 
outcomes. In cultural education programs, the relationship between 
knowledge -based program features (the quality of cultural content, cul- 
tural experience, intellectual focus, etc.) and program outcomes (the kind 
and quality of instructional materials, the content emphases of continuing 
education programs and curriculum development initiatives, etc.) is not 
linear. Rather, this relationship is multidimensional and discontinuous, 
revealed in oftentimes undiscovered or un-evaluated outcomes such as in 
instructional materials, classrooms, the network-building skills of program 
participants, or the focus of continuing education programs. Thus, it is 
necessary for an evaluator to take special care to disaggregate the various 
program elements and to construct ways to assess how knowledge-based 
program elements might affect program outcomes. 

Does the leader reveal an understanding of the value of multiple 
perspectives in the study of another culture? 

Multiple perspectives in the study of another culture are important 
because they help learners challenge conventional wisdom, official inter- 
pretations, and taken-for-granted ways of thinking. They are valuable 
because they present the world as a complex place where ambiguity is 
more at home than certainty. The following rich example best illustrates 
the benefits of looking at multiple perspectives. 

A one-week seminar for teachers in Alaska in 1994 was organized 
around the problem of a dramatic fall in prices in that state’s seafood 
industry — the source of the largest amount of non-government jobs. 
Fishermen from the United States complained that Japanese companies 
were illegally “fixing” the price of salmon. A sheen of racism appeared on 
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the surface of what was already a toxic brew — a situation ideal for examin- 
ing the traps of personal prejudice. In an attempt to encourage learners to 
reflect on the issue of whether or not Japanese companies were, in fact, 
engaged in the illegal manipulation of salmon prices, the seminar leader 
called for an examination of the differing perspectives of people involved: 

■ U.S. and Japanese salmon fishermen who blamedjapanese 
companies and Japanese salmon farmers for ruining their 
domestic markets; 

■ Japanese companies who perhaps profited from the misfortune of 
the Japanese and Alaskan fishermen; 

■ U.S. andjapanese salmon farmers who had gone bankrupt because 
of changes in the world salmon market; and 

■ U.S. andjapanese salmon wholesalers who had gone bankrupt for 
failing to sell their inventory in a falling market. 

The seminar leader also asked the participants to examine the validity 
of two allegations: that large Japanese and American seafood companies 
may be driving their fishermen into bankruptcy, and that unrestrained 
resource exploitation, motivated by greed and incompetence in both 
countries, will eventually decimate fish stocks everywhere. 

Swayed by reports in the local media that sympathized with the 
plight of the Alaskan fishermen, most of the participants were ready at the 
start of the program to blame the Japanese companies for the seafood 
industry’s economic problems. After careful study of the many perspec- 
tives of those involved, however, the class ended up divided between those 
who exonerated the Japanese and those who concluded that the Japanese 
companies were, in fact, engaged in illegal behavior. These same partici- 
pants, however, also believed that the Alaska salmon fishermen were blind, 
perhaps fatally so, to the mechanisms of the marketplace. They learned that 
Japanese fishermen had redoubled efforts to improve quality and thus 
remained competitive in a turbulent world market rather than simply rail 
against the perpetrators of the alleged market manipulation, as some of the 
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Alaskan fishermen had done. The result of the seminar was not that 
everyone came to the same conclusion, but that each participant made an 
informed decision while considering the perspectives of all involved. 

In searching for evidence that leaders have incorporated multiple 
perspectives into the study of a culture, a cultural education program 
evaluator can use a critical theory approach by asking: 

B Does the program invite participants to question the literature 
provided and to ask whose interests are being served by these 
materials or perspectives? 

■ If there is a program sponsor, does its agenda or ideology manifest 
itself in the program, in the leader’s behavior, in the readings, or in 
the program activities? 

B Are participants invited to reflect critically on the different per- 
spectives introduced by the program? 

fl Do leaders invite participants to reveal their own perspectives on 
issues and invite critical reflection on them? 

Does the leader emphasize problems and dilemmas as a 
way to develop participants 9 knowledge about their own 
and other cultures? 

Problem- or dilemma-centered programs are challenging for 
participants because they involve reflection, critical thinking, and other 
problem-naming and problem-solving skills. They invite participants to 
wrestle with ambiguity and with situations that stimulate the scrutiny of 
taken-for-granted assumptions. The previous example of how participants 
in a cultural education program in Alaska learned to see issues from more 
than one cultural perspective is illustrative also of a program with a focus 
on multiple problems. While some problems contained a cultural dimen- 
sion, particularly the suspicion of the Alaskan fishers that Japanese buyers 
were playing by different rules, other problems dealt with different fishing 
practices, attitudes towards consumers, the economics of commodities like 
seafood, changing global markets, and more. This example reveals how 
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certain powerful interests, including fishermen’s organizations, the mass 
media, arid local politicians, dominated the" debate by. blaming the invisible 
captains of the Japanese seafood industry. 

In searching for evidence of a problem-centered, critically-reflec- 
tive approach to program design and implementation, a cultural education 
program evaluator might ask: 

■ Does the program deal with one or more significant problems? 

■ Do the problems embody cultural issues? 

■ Whose viewpoints are represented in the ways the problems are 
framed? 

H Does the leader make an attempt to explore the role of power 
behind certain viewpoints and the way the dynamics of power can 
frame and distort how people construct meaning, especially 
meaning involving cultural differences and similarities? 

■ Do the problems address issues of local or regional interest to the 
program’s participants? 

■ Do the cases, examples, or materials used encourage participants 
to see complexities and ambiguities in the problems presented? 

Does the leader provide opportunities for participants to reflect 
upon themselves as bearers and creators of culture? 

A cultural education leader should be aware of the importance of self 
reflection as a source of learning about culture. Evaluators need to probe 
for evidence that the leader creates opportunities for participants to explore 
both the imprint that their culture has made upon them and also the way 
that they can borrow aspects from one or more cultures, creating a kind of 
hybrid culture. 

A good model of a self-reflective exercise is found in the case of a 
seminar leader who used a cultural artifact to spark a discussion in a group 
of Russian and American educators on aspects of American culture. 
Recognizing the 'imitations of attempting to generalize about an entire 
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culture from a single object, he handed them plastic soft drink containers 
that advertise movies and are sold in fast-food outlets in the United_States. 
The particular containers he used featured scenes from a movie popular at 
the time. The leader asked for one-word descriptions that would help 
people who had no knowledge of this culture develop working hypotheses 
about the people who created the object. 

The discussion focused on what the container was made of, what the 
material (plastic) said about the lifestyle and technological sophistication of 
the people of this culture, what the container was used for, and the mean- 
ing of the advertisements on the container. The analysis of the soft drink 
container revealed interesting generalizations about contemporary U.S. 
culture that seemed to surprise the participants. This led to lively discus- 
sions and conjectures about meanings of size, shape, color, and material of 
objects, as well as technology, lifestyles, gender roles, economic systems, 
and more. 

Indeed, when the leader observed that the use of plastics by this 
particular culture indicated a degree of “waste,” one of the participants, an 
economics teacher, noted that economists do not believe in the concept of 
“waste,” thereby delineating the beliefs of economists as a subset of the 
larger culture. This example reveals the importance of reflecting on the 
potentially loaded cultural meanings of concepts such as “waste” and 
“consumption.” In situations such as these, evaluators can determine the 
extent to which leaders have created opportunities for participants to 
become more mindful of their own habits of heart and mind and of appar- 
ent reality by making what is usually intangible more tangible. Thus, 
amorphous concepts like “mass culture,” “pop culture,” “Western cul- 
ture,” and “global culture,” become more tangible through the use of 
objects combined with critically reflective conversation about their 
meanings. 

Does the leader help participants apply cultural learning 
in their workplace settings? 

Evaluators should be concerned with the relationship between what 
is learned in the program and whether the leader helps participants apply 
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this learning to the workplace setting. Positive indicators might include 
program, goals that emphasize application of facts and knowledge, curricu- 
lum materials that are provided to participants that can be adapted easily to 
their work settings, sessions that model the teaching of certain concepts, 
discussions on how to present new ideas or materials, and participants’ 
statements that indicate how they will apply what has been learned in their 
work sites. 

Numerical ratings of the “usefulness” of each speaker or each session 
is one way evaluators can probe for evidence of the practical utility of 
various aspects of the program. An evaluator can also determine whether 
the participants recognize the leader’s emphasis on applied knowledge. 
Another way for an evaluator to assess the transferability of ideas is to ask 
participants to describe how they might teach certain concepts. Their 
written or oral explanations can indicate their grasp of concepts and how 
well they might be able to explain them to other learners. Whether the 
participants proceed to teach the concepts accurately requires classroom 
observation, a topic taken up in Chapter 4: Evaluating Program Outcomes. 



Communicative Disposition 

A third approach to the evaluation of cultural education leadership 
involves an assessment of leaders’ communicative disposition. A frequently 
overlooked feature of cultural education leadership, communicative dispo- 
sition involves the capacity of program leaders to communicate effectively 
across boundaries of race, class, and gender; to work sensitively with 
diverse colleagues and counterparts in other nations and cultures; and to 
prepare participants and cultural informants to learn from one another. 
Equally important is the leader’s ability to communicate well with partici- 
pants: to learn about their backgrounds, discover their assumptions about 
culture and cultural education, unearth their needs and hopes for the 
program, reconcile diverse interests, develop a group profile that repre- 
sents all participants, and create cultural learning occasions that are stimu- 
lating and challenging for all. 
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In this section, we will discuss how a cultural education program 
evaluator can assess the leader’s communicative disposition by using the 
following four questions: 

■ Is the program leader able to engage in culturally sensitive com- 
munication with people from a variety of cultures? 

■ Is the leader fully conversant with participants’ needs, and can he 
or she create cultural learning occasions that are stimulating for all 
participants? 

■ Does the leader prepare cultural informants to create learning 
opportunities for program participants? 

■ Does the leader prepare participants to interact with people from 
the target culture? 

Is the program leader able to engage in culturally sensitive 
communication with people from a variety of different cultures? 

In creating and implementing a cultural education program, the 
program leader must interact with a variety of people from a variety of 
cultures. They may include teachers, school administrators, scholars, 
professors, diplomats, businesspeople, artists, students, funders, and evalu- 
ators. They may differ in race, ethnicity, gender, age, nationality, and 
economic class. They may be conservative or liberal, rich or poor, native- 
born or immigrants, or from urban or rural areas. They may be partici- 
pants in the program, cultural informants who reveal the complexities and 
nuances of culture, or facilitators who help ensure that the program runs 
smoothly. The ideal leader will be able to engage in culturally sensitive 
communication with these very different individuals. 

Evaluators can look for evidence that the leader is mindful of alterna- 
tive forms of cultural learning, knowing, and doing; that he or she does 
not rush to judgment; and that he or she is comfortable with ambiguity. 
Further evidence that a leader can communicate sensitively includes a level 
head, a sense of humor, and an observant nature. While it is difficult to 
assess these qualities in a leader, a cultural education program evaluator has 
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a few options. First, he or she can gauge the nature of his or her own 
.interactions with- the leader and observe how; the leader interacts with v 
others. Second, the evaluator can examine program documents, which 
may include correspondence with cultural informants, missives to partici- 
pants, notes of telephone conversations, or reports to funders. Third, he or 
she can interview the leader and do spot checks by interviewing represen- 
tatives from each of the groups with which the leader communicates. 

Is the leader fully conversant with participants* needs , and can 
he or she create cultural learning occasions that are stimulating 
for all participants? 

One of the most important features of a leader’s communicative 
disposition is his or her ability to accurately assess participants’ needs. The 
assessment process involves a conversation between the leader in his or her 
capacity as program designer and the educators who participate in the 
program. Participants’ needs can be identified through a carefully crafted 
needs assessment process that involves the identification of espoused client 
needs on one hand, and other educational needs (Schon, 1983) — ones that 
may be determined by looking beyond the wholly legitimate but sometimes 
too narrowly constructed needs expressed by participants — on the other. 

Evaluators may ask program leaders how they determined what 
participants’ needs were. The previous chapter explained in detail the 
processes by which the leader and program evaluator can learn about the 
participants, their backgrounds, their assumptions about culture and 
cultural education, and their needs and hopes for the program. The chal- 
lenge for the evaluator is to determine how well the leader completed that 
process and then whether the leader used that information to create a 
program that met the needs of the participants. Questionnaires and partici- 
pant interviews provide useful evaluation data to answer this question. 

The evaluator can further ask; 

N Did the leader create a democratic, interactive climate in which 
participants felt free to make suggestions intended to improve 
program content and thus better meet needs, including ones that 
emerged as the program unfolded? 
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■ Did the leader ask participants to respond to the quality of guest 
speakers and activities, for example, through the use of daily 
evaluation instruments? 

■ Were participants asked to note the degree to which the day’s 
activities were helpful? 

■ Did the leader incorporate reasonable suggestions or requests into 
the program design? 

■ Did the leader change the original design to accommodate the 
espoused needs of participants? 

Does the leader prepare cultural informants to create learning 
opportunities for program participants? 

As the previous chapter suggested, the role of cultural informants in a 
program is crucial, and their ability to work with the program leader to 
create learning opportunities for participants may make or break a pro- 
gram. Depending cn the nature of the program, a cultural informant 
might deliver a lecture; lead a discussion with participants; arrange a site 
visit; conduct a guided tour; give a presentation, demonstration, or perfor- 
mance; or host a participant in his or her home. The ability of the infor- 
mant to create appropriate educational occasions for participants will rely, 
in large measure, on the communication between the informant and the 
program leader. 

Prior to the start of the program, the leader can prepare the cultural 
informant by communicating as much as possible about the program and 
its participants. Pertinent information might include: the purpose of the 
program, the backgrounds and interests of the participants, their level of 
cultural knowledge and language ability, how the informant’s session fits 
into the larger program, what the leader and participants hope to accom- 
plish in the session, and how the participants will use the information and 
insights gleaned in the session. The Group Profile described i chapter 2 
will contain much of this information and will likely prove an indispens- 
able document to the cultural informant. 
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The evaluator can look for evidence that the program leader has been 
in frequent contact with his or her cultural informants and -has provided - ; _ 
them with the necessary information. In addition to interviewing both the 
leaders and the cultural informants, the evaluator may also wish to exam- 
ine correspondence, records of conversations, or the Group Profile. 
Participants, too, are good sources of information as they will have opin- 
ions on whether a cultural informant understood their needs and created 
an appropriate opportunity for cross-cultural learning. 

In an example of a cultural education seminar with a five-week 
component in Japan, the extensive efforts of the program leader to prepare 
the Japanese cultural informants paid off handsomely. The twelve U.S. 
participants were elementary school teachers, and the program gave them 
the opportunity to live for a week in the home of a Japanese elementary 
school teacher and accompany him or her to school every day. Well before 
the trip to Japan, the program leader required each American participant to 
complete a “Homestay Profile,” separate from the information collected 
for the Group Profile. The Homestay Profile included both professional 
information — grade level taught, subjects taught, specialties, and profes- 
sional interests — and personal information — family, hobbies, food prefer- 
ences, past experience abroad, etc. The program leader’s contact in Japan 
matched each U.S. participant with a Japanese counterpart using the 
Homestay Profile. He also gave a copy of the profile to the host family and 
the host school. 

As is frequendy the case, the evaluator found that the one-week 
homestay was considered the highlight of the trip by most participants. 

The participants were amazed by the quality of the matches. Indeed, using 
the Homestay Profile, the leader’s contact in Japan was able match an 
American music teacher with a Japanese music teacher, a teacher interested 
in children’s literature with a Japanese teacher who was an amateur story- 
teller, and a teacher interested in setting up a pen pal exchange with a 
Japanese teacher who wished to do the same. 

The participants were also thrilled that they were able to accomplish 
so many of their goals while visiting the homes and schools. For example, a 
physical education teacher participated in after-school sports activities; a 
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teacher interested in environmental education accompanied a fifth grade 
: classona field trip to. a local recycling station; and an art teacher attended a 
weekly pottery class with a member of her host family. The participants 
attributed this success to the kindness and generosity of their hosts, but the 
evaluator also attributed it to the thoroughness of the Homestay Profile. 
The evaluator concluded that the Japanese — both the families and the 
school personnel — understood what the U.S. teachers hoped to accom- 
plish during the week and, consequently, were able to create superb 
learning opportunities for the Americans. The evaluator gave high marks 
to the leader for preparing the Japanese cultural informants well. 

Does the leader prepare participants to interact with people 
from the target culture? 

While it is important for the leader to prepare the cultural informants 
to create learning opportunities for the participants, it is equally important 
to do the reverse: prepare the participants to interact with the cultural 
informants. This involves preparation of two kinds. First, the evaluator can 
look for evidence that the leader communicates as much as possible to the 
participants about the cultural informant. For example, suppose the par- 
ticipants in a Japan seminar are going to hear a lecture from a visiting 
Japanese professor on the justice system in Japan. The evaluator would 
want to note whether the participants received such information as: the 
background and interests of the professor, how his lecture fits into the 
larger program, and what the leader hopes the session will accomplish. If 
the professor’s lecture is to be interpreted, the evaluator can also look for 
evidence that the participants were given instruction on the etiquette of 
using an interpreter. 

The second kind of preparation that evaluators can look for involves 
the extent to which the leader equips participants with the knowledge, 
skills, and dispositions to interact sensitively and effectively with someone 
from the target culture. Especially if the program includes a study abroad 
component or immerses the participants in a culturally unfamiliar setting, 
participants may clamor for knowledge about the culture to avoid embar- 
rassing blunders or inappropriate behavior. Important cultural knowledge 
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might include modes of acceptable dress, standards of cleanliness, or 
proper table manners. Skills that maybe required to move comfortably in 
another culture might include how to ask questions, demonstrate respect, 
indicate disagreement, or recognize when you have made someone un- 
comfortable with your words or actions. Dispositions that a leader may 
wish to instill in participants might be the ability to reflect on the mean- 
ings of words and actions, to suspend judgment until more facts are 
known, or to resist the temptation to oversimplify or overgeneralize. This 
kind of preparation will help learners come to a more sophisticated under- 
standing of the culture and may help smooth the way for participants as 
they traverse the oftentimes rocky road of cultural adventuring. 

For the evaluator, evidence that the leader has tried to prepare the 
participants to function in another cultural context might be found in 
readings, lecture topics, discussions, activities, or written directives from 
the leader to the participants. The evaluator might interview the leader, 
participants, and/or cultural informants, all of whom will have opinions on 
the adequacy of cultural preparation. When the preparation is insufficient, 
the consequences can be hurt feelings, damaged reputations, and devas- 
tated relationships. An example of such a case occurred in a program in 
Thailand, during which a group of American educators treated a 
university’s guest house as a hotel and left it, in the eyes of the Thai hosts, 
in a mess. The incident, the cause of which was apparently a failure on the 
part of the American leader to specify expectations for staying in the 
facility, left the Thai hosts angry, the American leader embarrassed and 
apologetic, and the participants defensive. In the end, the American leader 
was unable to mend the relationship, and the Thai university terminated 
the association. The program’s evaluator was able to find documentation 
of this incident in written correspondence, interviews with the Thai colleague 
and the American leader, and participant responses to questionnaires. 

When cultural preparation is good, however, the potential for cross- 
cultural learning is magnified greatly, with implications for both personal 
growth and professional practice. An example of successful preparation 
took place in a cultural education program for U.S. teachers with a two- 
week seminar in Japan. The leader tried to prepare the participants for the 
trip by devising a number of group activities that the leader hoped would 
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simulate how the participants would be required to operate as a group in 
Japan. Attention was paid to group dynamics and decision-making by : 
consensus. The leader asked the participants to reflect on implicit inequi- 
ties in power relations among members in a group and to examine their 
assumptions about others in the group who may be perceived as having 
more or less power. 

After the seminar, one participant wrote that he became keenly 
aware of the tensions created by individual behaviors when the group 
attempted to operate as a group of Japanese might operate. “I became 
much more aware of the difficulties involved as 12 individualists tried to 
make decisions that we supposedly made in the best interests of the group. 
And I became much more aware of my own behavior, sometimes passive, 
sometimes aggressive — often annoyingly and surprisingly so — as I learned 
to watch myself in a group context. I now have a better idea of the behav- 
iors that help a group accomplish its goals, and when to assert myself in a 
purely individualistic way” (Barry, 1993b, p. 32). This participant also felt 
that the experience made him better able to understand the constraints 
under which the Japanese teachers he encountered might be working. By 
helping this participant reflect on the cultural origins of group behavior, 
the program leaders helped him not only to view the world from the eyes 
of another, but to expand his own repertoire of responses in group settings. 

At the conclusion of a cultural education program, the evaluator can 
ask the following additional questions of participants to judge the success 
of their leader in promoting culturally sensitive communication: 

M Did you encounter confusing or anxiety-provoking situations as 
you studied in the culture? What, if anything, helped you deal 
with those situations? 

■ Were you invited by program leaders to compare cultures and to 
reflect on larger cross-cultural themes? 

■ How could you have been better prepared to meet the challenges 
of cultural exploring? 
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Community Building 

In the design and implementation of cultural education programs, 
program leaders v 'I, of necessity, work in varied communities of cultural 
teachers and learners. In fact, they will be unable to orchestrate the many 
elements of a sophisticated cultural education program without help from 
diverse colleagues in other nations, cultures, or communities. Sometimes 
the leader will simply tap into the resources of a community. A much 
more demanding role, however, is that of cultural bridge-builder who 
links people across boundaries of nationality, race, class, gender, profes- 
sion, etc. When leaders function as community builders, they foster 
“cultural contact zones,” enclaves of intercultural reflection, association, 
and repertoire-building within which mutual understanding and perspec- 
tive-taking function as both theory and practice (Bateson, 1994; Eddy, 
1996; Finkelstein, 1992, 1995). 

Evaluators can assess leaders’ community building capacities by 
asking the following four broad questions: 

Si Does the leader engage in community building? 

a Is the leader involved with a varied network of cultural 
informants? 

B Does the leader have a systematic and objective approach to 
selecting program participants? 

B Does the leader have a plan for maintaining contact with partici- 
pants after the program? 

Does the leader engage in community building? 

Building learning communities or “cultural contact zones” for 
reflection, perspective-taking, and sharing of cultural autobiographies 
requires the leader to consciously foster certain dispositions. The differ- 
ence between communities of racists, fascists, or groups advocating vio- 
lence against the U.S. government, and communities of cultural educa- 
tional communitarians is that the latter groups champion what was re- 
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ferred to earlier as. “democratic dispositions,” meaning attitudes and skills 
that feature respect for difference and the ability to listen, show respect for 
people with different views, discuss disagreements thoughtfully, and make 
compromises. 

As Paulo Freire and others (Brookfield, 1995; Freire, 1993, 1994) 
have pointed out, democratic teaching and learning does not mean that 
educators cease to speak authoritatively, that they pretend to be like their 
students, or that they create laissez-faire atmospheres of intellectual relativ- 
ism in which anything goes and fine distinctions disappear. What it does 
mean is that philosophical contradictions and practical tangles inherent in 
the democratic ideal are acknowledged, educators and program partici- 
pants treat each other with respect, educational processes are open to 
genuine negotiation, and spaces are created in which all voices can speak 
and be heard. 

Evaluators can look for this kind of community building in the 
personal visions of cultural education leaders and in the documents that 
these leaders generate. Documents may contain the ground rules for 
shaping a community of reflection. Indicators of how well leaders succeed 
can be found in observations and participant comments on the program. 
Evaluators might also ask: 

■ Is the leader aware of the community-forming role of cultural 
education leaders? 

■ What is the leader’s definition of community? 

■ Does the leader recognize any of the “democratic dispositions,” 
and are they evidenced in the way the program is conducted? 

Is the leader involved with a varied network of 
cultural informants? 

Often, when working in another culture, little can be accomplished 
without proper introductions or personal connections. For this reason, a 
leader must maintain good relations with a diverse network of cultural 
informants in order to create a sophisticated cultural education program — 
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the more elaborate. the program, the greater the need for a broad spectrum 
of informants. A local cultural informant network might consist of stu- 
dents from the target country studying in the U.S., visiting academics, 
diplomats and their families, business leaders, or fellow Americans with 
substantial knowledge, of the target culture. A national network might 
include other program directors, heads of professional and cultural associa- 
tions, scholars, and politicians. To conduct a program with a study abroad 
component, a leader might require an international network of teachers, 
students, parents, school principals, university professors, government 
officials, sister city representatives, artists, expatriates, businesspeople, 
museum educators, or community activists. 

Evaluators can assess whether the leader maintains an adequate 
network of cultural informants and, if so, the level of its quality. Assess- 
ments of the quality of the network are bound to be subjective. Leaders, 
themselves, may realize the shortcomings of their network as typified in 
comments such as, “I need to cultivate someone higher up in the Ministry 
of Education if I am ever going to get beyond the official line.” Partici- 
pants, too, may be delighted or disappointed at the leader’s ability to 
connect them to people and opportunities. Evaluators can spot these 
feelings in participant comments such as “I can’t believe she was able to get 
us backstage at the theater,” “What a pleasure it was to hear about the 
research straight from the author,” or “I wanted to visit a retirement home, 
but the leader didn’t know anyone there.” 

Evaluators may wish to have participants evaluate the quality of 
expert presentations or the usefulness of an informant’s session as a way to 
assess the quality of informants and the leader’s network. The perceived 
quality of experiences — for example of homestays organized by the infor- 
mant — may provide valuable evaluation data. Participant complaints 
about the quality of homestay experiences, for exampile, might suggest a 
problem with the leader’s network. 

Evaluators may also look for evidence that leaders understand the 
give-and-take of cultivating a network of cultural informants. Program 
leaders who attend to the needs of the cultural informants as well as the 
needs of their own program ensure healthy, reciprocal relationships. For 
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example, the leader may reciprocate by offering to give a lecture, coordi- 
nate a visit, or serve as a host to international visitors. The evaluator can 
interview the leader and cultural informants, or examine correspondence 
between them, for indications that the arrangements are mutually beneficial. 

Leaders who are successful community builders are generous when it 
comes to sharing their professional contacts with participants, often in the 
belief that sharing enlarges and enriches a community and creates new 
communities. Evaluators can be on the lookout for ways in which the 
leader shares his or"her contacts, not only with participants, but by linking 
one cultural informant with another. The participants, too, may have their 
own contacts, and the evaluator can look for evidence of the ways the 
leader utilizes resources offered by the participants. 

Does the leader have a systematic and objective approach to 
selecting program participants? 

When leaders bring together a group of educators for a cultural 
education program, they create a new community of cultural learners and 
teachers. For this reason, the selection of program participants — the 
members of that community — can be decisive. No matter how good the 
program, if the participants are disinterested or unmotivated, the program 
results will be disappointing. If, however, the participants are excited about 
the possibilities of cultural education, committed to the program and its 
goals, and eager to play an active role in the community of cultural learners 
and teachers, the program is much more likely to achieve successful and 
enduring results. 

The selection of participants requires a relatively significant block of 
time to publicize the program and to learn as much as possible about the 
audience from which participants can be drawn. It is important not to be 
rushed into selection decisions because of inadequate recruitment time. 
There are many cases in which poor planning and political pressure to get 
“bodies in the program'’ led to unfortunate choices, disappointing pro- 
gram outcomes, and thwarted or dysfunctional professional communities. 

Evaluators can assess how leaders use professional contacts and 
networks in the selection process. For example, do they call upon curricu- 
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lum directors and school principals?-If the program targets college faculty, 
does the leader know the appropriate deans, department chairs, or others 
who might ensure that written materials get the widest possible circula- 
tion? Is there evidence of a “marketing plan,” a systematic effort to target a 
particular audience and to reach all the potential members of that audi- 
ence? Evaluators should take care to note unfocused, last-minute recruit- 
ment efforts and look for indications that too much emphasis is placed on 
personal contacts and references. 

Evaluators can look for evidence that leaders use objective criteria to 
select participants. Depending on program goals, selection criteria might 
include curriculum development experience, teacher in-service presenta- 
tions, talks before civic groups, professional publications, team teaching, 
presentations at conferences, etc. Leaders may also place an emphasis on 
the answers to such questions as: “Why did you apply for this program?” 
“If selected, how will you apply what you learn?” and “What is your 
experience in curriculum development?” If there are established selection 
criteria, it is important for the evaluator to match the criteria against the 
resume and applications of the selected participants. It is common in cases 
in which program outcomes were not realized to find discrepancies be- 
tween program goals, selection criteria, and participant backgrounds. 
Because of the cost of continuing professional education, especially those 
with a study abroad component, funders place substantial weight on a 
program leader’s ability to select good participants. 

Does the leader have a plan for maintaining contact with 
participants after the program? 

Another important dimension to community-building is the leader’s 
efforts to remain in contact with and to provide continuing benefits to 
participants after the formal part of the cultural education program has 
ended. Evidence of a continuing commitment to community building can 
be found in interviews with program leaders and in official documents 
generated by the program, such as instructions for helping community 
members stay in touch, newsletters, and records of personal phone calls. 
Further evidence might be the leader’s efforts to create follow-up activi- 
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ties, such as meetings to share experiences and insights, or additional joint 
projects, such as grant-writing, curriculum development, or conference 
presentations. In general, evaluators should look for evidence of specific 
intentions and actions leading to the creation of spaces that allow commu- 
nities of cultural reflection to grow and flourish. 



On the Road with Cultural Education 
Leaders: A Sample Evaluation 

In this section we provide an in-depth example of how an evaluator 
might go about assessing a cultural education program. This is the evalua- 
tion of a hypothetical professional development program that is likely to 
become more commonplace in the future: an educational experience 
involving a brief stay in another country. The complete evaluation would 
cover the four characteristics of cultural education leaders that are described 
in this chapter. In this excerpt, however, we focus on the concept of com- 
munity building, and specifically on participant selection, the leader’s net- 
work of cultural informants, and opportunities for ongoing networking. 

The Program 

A curriculum specialist in a public school district decided to take a 
group of high school teachers to Japan to study the culture and bring back 
materials and lessons. She designed a program in which the participants 
studied Japanese culture in evening classes throughout the spring and then 
traveled to Japan in the summer for a two-week seminar. Funding for the 
program came from a private foundation that supports the broad goal of 
increasing knowledge about Japan andjapanese culture among U.S. 
citizens, especially those involved in education at the precollege level. 

The participants first went to Tokyo, where they visited temples, 
toured the Japanese parliament, attended a kabuki performance, and spent a 
day in a Japanese high school. They then proceeded to Hokkaido, the 
northernmost island of Japan, where they observed geographic, economi/, 
and social contrasts between rural and urban Japan. In Sapporo, 
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Hokkaido’s largest city, they visited local government officials, observed 
classes at a high school, and spent two nights with Japanese families. 
Throughout the trip, the participants heard lectures from Japanese experts 
on topics that interested the group, including religion, traditional theater, 
school-to-work training, and the consumer rights movement in Japan. 

The program also set aside several days for pursuing individual interests. 

Community Building: Participant Selection 

The evaluator felt that one of the program’s strong points was the 
quality of the participants and, after examining the leader’s selection 
process, determined that there were a number of things the leader did well. 
First of all, she hired the evaluator early and, after consulting with other 
program leaders around the country with experience organizing study 
trips for teachers, developed a fairly extensive strategy for participant 
selection. In addition to writing essays on the theme of how they would 
make use of the travel opportunity, applicants were also asked to submit 
detailed individual profiles for eventual use in matching the teachers with 
Japanese host families. 

The evaluator commended the leader for starting the recruitment 
pre jess well in advance of the trip. The leader realized that adequate lead 
time is important so that busy teachers can plan ahead for their summer 
vacations and so program staff can make the trip arrangements with their 
equally busyjapanese counterparts. In the evaluator’s judgment, advance 
planning indicated a cultural sensitivity that recognized the need on the 
Japanese side for time to prepare Japanese hosts to receive American 
visitors and thus avoid embarrassing blunders that can strain or break the 
all-important human bonds that are often key to the success of these 
cultural encounters. 

Given limited funding, the program leader wanted to select partici- 
pants who would maximize the investment — teachers who showed com- 
mitment to introducing the study of another culture into their teaching 
repertoire or to improving what they were currently teaching. Her criteria 
for participant selection included: a personal philosophy on the teaching of 
other cultures, prior international travel experience, potential for curricu- 
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lum leadership in their school, curriculum development experience, .. 
quality of teaching as confirmed by references, and an interest in Japan. 
Using this information, the leader and a small team of advisors were able to 
select a group of teachers who were enthusiastic about cultural education 
and who were eager to form a community of cultural educators. 

Evaluator’s Recommendations: The evaluator could find little 
fault with the leader’s approach to participant selection. The evaluator was, 
however, able recommend that the leader try a team application approach. 
This approach requires candidates to apply in teams of two to four people, 
usually from the same school (or school district for larger programs). The 
team should demonstrate a willingness to work together and support one 
another upon completing the program. The idea behind this approach is 
that a single educator trying to effect changes in cultural education can 
become isolated and discouraged. A team of educators, however, having 
gone through the same cultural education program, can share ideas, 
collaborate on projects, and encourage one another. 

Community Building: A Network of Cultural Informants 

The leader’s network of cultural informants in this example included 
cultural experts, schools, government officials, and an organization that 
coordinated the two-night homestays. The central questions for the 
evaluator to ask were: How extensive and broad is the leader’s network? 
and What is the quality of that network? 

The first question could be answered by examining the scope of the 
network. In this example, the scope seemed rather extensive: contacts in 
education, government, and local communities who provided access to 
learning experiences that went well beyond superficial touring. From an 
evaluation standpoint, having one of these resources — contacts in the 
Japanese Ministry of Education — is good, but having an additional re- 
source — access to schools where participants can get a different perspec- 
tive — is better. 

Answering the second question — determining the quality of the 
network — was rickicr and so the evaluator relied, in large part, on reac- 
tions from participants. Interview questions designed to approach subjects 
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indirectly by asking logistical questions and -by having participants describe - 

the benefits from and usefulness of specific events such as school visits, 
presentations by cultural experts, and homestays yielded participant judg- 
ments concerning the quality of the leader’s network of cultural informants. 

In this case, participants gave the program high ratings, except in a 
couple of cases in which special needs were not met and where visits with 
Japanese families yielded unintended results, such as a late night drinking 
party where the male host made remarks that offended one of the partici- 
pants. The several lectures by Japanese experts — on religion, traditional 
theater, school-to-work training, and the consumer rights movement in 
Japan — were given high marks, suggesting considerable skill on the 
leader’s part in finding high-quality Japanese informants to match a broad 
array of requests. A few participants mentioned that the talk by the Minis- 
try of Education official was better than they had expected. They had 
anticipated a boring one-hour recitation of the official line — what they 
had already read in government publications — but instead they found their 
discussion with him open and candid. 

Evaluator’s Recommendations: The program leader seemed to 
understand that the quality of contacts on the Japanese side can determine 
the success of the program. In this example, one of the things participants 
wanted to learn about was Japanese consumer characteristics. The leader 
asked a Japanese businessman friend to introduce her to Japanese infor- 
mants at the retail and wholesale levels in order to arrange briefings about 
demand for certain products, presentation, pricing, and so on. The leader 
already knew about Japanese government organizations that provided print 
and video information in English. 

Beyond offering official government perspectives on consumer and 
trade issues, the evaluator suggested that the program ought to include 
additional perspectives, such as Japanese retailers who complain about 
Japanese government regulations and consumer groups who allege that 
regulations keep consumer prices high and keep competing products, 
especially those from foreign countries, from getting a foothold in the 
Japanese market. Leading participants to the realization that there is con- 
siderable diversity of opinion about the relationship between the role of 
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_ government and the behavior of markets in Japan will help to challenge 
stereotypes of docile, compliant Japanese citizens and an omnipotent, 
single-minded Japanese government bureaucracy. 

Because the leader mentioned an interest in adding more about 
Japanese education to the program in the future, the evaluator stressed the 
importance of using informants who can reveal the diversity of Japanese 
schools. Japanese schools, like those in America, come in many varieties, 
are situated in both rich and poor neighborhoods, have varying degrees of 
discipline and academic performance problems, and, in some cases, have 
diverse student populations, including minority buraku people and recent 
immigrants from China and South America. To experience this diversity is 
to behold a much more complex picture ofjapanese education than can be 
ascertained from perhaps more readily available but less varied descriptions. 

The evaluator found one complaint from the program participants 
about the scope of the leader’s network of informants in Japan. Some 
participants felt that the leader had failed to take seriously offers to use 
their Japan contacts to further enrich the program. As a result, the evalua- 
tor recommended that, to the extent possible, the leader take into consid- 
eration the knowledge, experience, 3nd contacts that participants bring to 
the program. 

Community Building: Opportunities for Ongoing Networking 

Once they help participants create a professional community of 
critical and cultural reflection, leaders should then help sustain that com- 
munity by creating spaces for dialog and collaboration after the formal 
program has been completed. In this aspect ot community building, the 
leader of this study program in Japan received the lowest marks. The leader 
conducted one final workshop in late summer and, in an “I survived the 
Tokyo subways” graduation ceremony, brought the project to a humorous 
close. The participants continued to work on their Japan projects, excited 
about the prospect of trying out new ideas and materials when classes 
began in the fall. Meanwhile, the leader, a busy curriculum specialist, 
moved on to her next project. The leader and the participants, all working 
in the same school district, often ran into one another and continued to 
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enjoy the camaraderie that had-developed during their cultural . adventures. 
The leader, however, made no further attempt to bring the participants 
together or build on the community she had created. 

Evaluator’s Recommendations: After a program in which the 
leader excelled in community building, the evaluator was disappointed 
when the leader considered her work finished at the conclusion of the 
formal program. The evaluator recommended that the leader reconsidc r 
her role as a catalyst for ongoing networking. The evaluator’s suggestions 
for promoting further networking included: updates on programs and 
projects that are ongoing; a mechanism for encouraging requests from 
former participants that make additional use of the leader’s contacts and 
networks; opportunities for collaborative publications or curriculum 
development; joint proposal-writing; opportunities for former participants 
to serve as faculty in future education programs; collaborative development 
of conferences, seminars, and meetings; a newsletter that collects and 
disseminates new ideas and best practices; and/or the formation of new 
professional organizations and associations. The evaluator saw potential in 
any of these ideas and encouraged the leader to preserve and deepen the 
community that formed as a result of the program. 



Conclusion 

Educators who consciously involve the study of culture in formal 
curriculum can be considered cultural education leaders. This chapter 
argues that the four characteristics of cultural education identified in 
chapter 1 — leaders’ vision, knowledge, communicative disposition, and 
capacity for community building — provide criteria for evaluating cultural 
education leaders in their program contexts. These criteria are consistent 
with various attempts by scholars from different disciplines, including 
education, to theorize about leaders and their organizational settings. 

Better leaders and programs, along with a wiser expenditure of 
money, may not be the ultimate goals of evaluation. Above all, through the 
cultural education programs they develop and refine, leaders and evalua- 
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tors contribute to_ the creation_of new educational spaces in emerging 
borderlands of awareness and experimentation where old, often comfort- 
ing assumptions about sense of place and identity are confronted and 
reconfigured. The challenge of our times, according to Bhabha (1994), is 
to focus not on the “exoticism of multiculturalism or the diversity of 
cultures” (p. 38) but on spaces and processes that facilitate the articulation 
of culture’s hybridity, a dynamic, transactional process of mutual influence 
in which it is possible to “elude the politics of polarity and emerge as the 
others of ourselves” (p. 39). 

Houston Smith (1995) put it this way: “Anyone who is only Japanese 
or American, only Oriental or Occidental, is but half human. The other 
half that beats with the pulse of humanity has yet to be awakened” (p. 13). 
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■ I 

JLn this chapter, we will explore ways to evaluate 
the outcomes of cultural education programs. Unlike 
more traditional approaches to the field of cultural 
teaching and learning in which evaluators generally aim to link program 
evaluation to transformations in cultural attitudes and adaptability (Kelley 
Sc Meyers, 1993; Kohls Sc Knight, 1994; Pusch, 1979; Hess, 1994; Stewart 
Sc Bennett, 1991), we have chosen to take a different approach. We have 
linked our assessment of program quality to concrete, analyzable, and/or 
measurable behavioral outcomes, as in the nature and quality of instruc- 
tional materials prepared by program participants, the cultural sensitivity of 
the resulting student assessment measures, and the plans for participants’ 
own professional development and that of others. 

In our view, behavior-focused evaluation provides evaluators with 
concrete ways to assess the effectiveness of programs as well as reveal the 
effects of attitudes. As we see it, programs are most effective when a 
transformation in participants’ thinking translates into behavior that 
improves teaching about other cultures, helps schools guide students to live 
successfully in a multicultural world, and assists educators with new possi- 
bilities for communication among culturally disparate groups. 

In this chapter, we suggest frequently overlooked places in which an 
evaluator can discover evidence of participants’ changed behavior. We 
propose tools that evaluators might employ to document results of pro- 
grams, such as questionnaires, interviews, observation, and document 
analysis. We also recommend questions that evaluators can use to probe for 
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outcomes. We often suggest a comparison between educators’ behavior 
prior to their participation in a cultural education program and their " 
behavior after its completion. Pre- and post-progrv -n evaluations are useful 
as are comparisons of participants’ behavior to that of non-participating 
control groups. Also, because it takes time to change attitudes, develop 
quality materials, and build support for educational reform, the full results 
of cultural education programs may not be apparent for months or even 
years. For this reason, where possible, we encourage long-term evaluation. 

Thus far in this book, we have been addressing four features of 
cultural education programs: cultural vision, knowledge, communicative 
disposition, and community building. In this chapter, we suggest that 
evaluators look for evidence of participant outcomes in the same four 
areas. We proceed on an assumption that graduates of effective cultural 
education programs will have learned to integrate cultural dimensions into 
the initiatives that they lead, whether they proceed in spaces as small as 
individual classrooms or as large as international arenas. Evaluators may 
find evidence of change in such places as the classroom, testing procedures, 
instructional materials, communication styles, and education policies. The 
remainder of this chapter encourages evaluators to examine participants’ 
behavior and is organized around participants’ cultural vision, knowledge, 
communicative disposition, and community building. 



Cultural Vision 

When participants of cultural education programs return to their 
workplaces, the activities they promote will reflect their view of culture. 

In the attempt to discover the vision on which program participants build 
their cultural repertoires, evaluators need to document the reasons educa- 
tors offer for attending to culture in the workplace. A staff development 
supervisor’s vision might be expressed in the cultural content of profes- 
sional development organized for the school district. A curriculum super- 
visor, on the other hand, might reflect his or her vision of cultural study in 
the development and content of new curriculum units, while a media 
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center specialist might exhibit cultural vision in the selection of new 
books, videos, or computer programs. 

The cultural visions of teachers, however, are for the most part 
reflected in their behavior in the classroom, in such areas as the materials 
they choose and the methods they use to evaluate student performance. In 
this section, we explore ways in which evaluators can discover and docu- 
ment the cultural visions of teachers as they are revealed in classroom 
behavior. To this end, evaluators might use interviews, questionnaires, 
analysis of materials, and classroom observation. We offer the following 
sample questions as tools to help unearth teachers’ cultural visions, not 
only in what they say but also in what they do. Slight modifications of the 
questions would make them appropriate for educators who are not class- 
room teachers. 

■ How does the teacher frame cultural goals? 

H How does the teacher filter information and choose content? 

■ What instructional materials does the teacher choose? 

■ How does the teacher teach the content he or she has chosen? 

■ How does the teacher evaluate student performance? 

How does the teacher frame cultural goals? 

Teachers are familiar with articulating goals for lessons, particularly 
ones that address knowledge or skills such as “to write individual and 
group haiku,” “to discuss Japan’s natural resource scarcity,” or “to become 
familiar with the history ofjapanese family crests.” After participating in a 
cultural education program, however, teachers may put considerable 
thought into the framing of cultural goals as well. 

By the term cultural goals, we mean goals that seek to reveal the 
existence of culture and its effects on ways of being and believing. Cultural 
goals can be framed in a variety of ways. For example, teachers may intro- 
duce the study of another culture to help students see the world through 
the eyes of another and understand that their world view is not universally 
shared. Teachers may want students to understand that there are many 
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different perspectives in the world and that those differences need to be 
understood, respected, and harmonized. They may invite students to study 
another culture in order to reflect on their own backgrounds. They may 
encourage students to learn to get along with people who possess different 
customs, habits, and beliefs. Teachers may try to eliminate students’ 
cultural stereotypes, teach them to value diversity, or help them to be less 
judgmental concerning the superiority or inferiority of other cultures. 

For an evaluator, the richness of this frame becomes a way to judge 
the cultural dimension of a teacher’s goals and purposes. Some teachers 
will be able to articulate their cultural goals clearly and concisely, especially 
if the cultural education program in which they participated encouraged 
reflection on the reasons for making culture a topic of study. Others may 
only know that they want to teach about other cultures but have vague 
reasons as to why. Interview questions such as “Why do you want your 
students to learn about other cultures?” “How will the study of other 
cultures help your students in today’s world?” or “What do you want your 
students to understand through the study of another culture?” may be 
particularly helpful in discovering teachers’ cultural goals. 

How does the teacher filter information and choose content? 

In every educational setting, at every moment in time, educators 
serve as cultural gatekeepers. They filter information about other cultures 
and choose what content to include or exclude in their lessons. For an 
evaluator, the capacity of program participants to articulate the reasons for 
their choices constitutes an important, if often ignored, reflection of the 
sophistication of their cultural vision. 

For example, teachers may choose content to present a complex view 
of a particular culture that is being studied, seek to portray sensitively the 
customs and beliefs in that culture, or choose content that dispels com- 
monly-held stereotypes. They may understand that culture is too complex 
to come across in one point of view and therefore strive to reflect multiple 
political, economic, aesthetic, and social perspectives from within the 
target culture. They may seek content that reveals regional variation as 
well as national consensus, minority group viewpoints as well as majority 
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— group perspectives, and current as well as historical perspectives. They- 
might organize information and concepts in such a way as to encourage 
comparisons with other cultures on larger themes such as child rearing 
practices, the global environment, social justice, or conflict resolution. 

Evaluators may use questionnaires and in-depth interviews to ask 
teachers directly about the content choices they make when teaching about 
another culture. Even more revealing, however, may be an analysis of 
materials used in the classroom that centers on what cultural content is 
included and what is excluded. Teachers may not be able to choose their 
textbooks, but they will consciously or unconsciously demonstrate then- 
cultural visions in their choice of content as revealed in lesson plans, 
supplementary materials, readings, videos, objects, and guest speakers used 
to portray another culture. 

We recently evaluated a workshop organized by an elementary 
school curriculum supervisor who had completed a cultural education 
seminar. The workshop was on Japanese education and the supervisor- 
defined a content focus that, in our view, revealed a nuanced concept of 
culture. Prior to the workshop, we interviewed her and analyzed her 
workshop schedule, choice of speakers, and handouts. The title of the 
workshop was “Japanese Culture through Education: The Perspectives of 
Teachers and Parents.” From the title alone, we could tell that she sought 
to offer more than one perspective onjapancse education. The 
supervisor’s choice of speakers reinforced her emphasis on multiple per- 
spectives. One speaker was an American woman who had taught in both 
Japan and the United States and entitled her talk, “Inside Japanese El- 
ementary Schools: The Teacher’s Perspective.” The second speaker was a 
Japanese man whose family had lived in both Japan and the United States 
and whose children had attended schools in both countries. His talk was 
entitled, “The Relationship between Home and School in Japan: My 
Family’s Experiences.” The handouts, too, reinforced the importance the 
supervisor placed on presenting the views ofjapanese as well as American 
scholars. This evidence, taken together, demonstrated to us that this 
supervisor was making conscious choices as she Filtered information on 
Japanese education and selected content, speakers, and readings that would 
reveal a complex view of culture. 
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What instructional materials does the teacher choose? 

A cultural education program has the potential to have a profound 
impact on the instructional materials a teacher uses in the classroom, A 
program may introduce participants to previously unknown materials, 
encourage reflection on and critique of existing materials, or inspire the 
creation of new ones. The evaluator may assess the effect of the program 
by attending to the level of sophistication of the instructional materials that 
the teacher chooses. 

First, the evaluator might explore evidence of good choices in the 
accuracy of the instructional material. Are the facts correct? Is the infor- 
mation up-to-date? Does it oversimplify the culture to try to make it 
understandable? Second, the evaluator might attend to the cultural tone of 
the material. Does it contain stereotypes or loaded words? Does it make 
the culture seem exotic or inscrutable? Does it excessiveh praise or criti- 
cize the culture? Third, the evaluator might explore the importance the 
teachers places on perspective. Who wrote the material? What point of 
view is included and what is excluded? Fourth, the evaluator might exam- 
ine how the teacher uses objects to teach about culture (Eder, 1995). What 
does the object tell about who made/owned it? What aspects of culture 
does it reveal? Finally, the evaluator might analyze the variety and combi- 
nations of materials for their reflection of multiple perspectives within the 
target culture. 

In our own experience with Japan-related instructional materials, we 
have discovered examples that illustrate the vast difference between mate- 
rials that promote stereotypes and outdated views of a culture, and those 
that provide a nuanced look at a culture from an insider’s perspective. 
Object learning kits, also called “artifact trunks” or “discovery boxes,” 
contain objects from another culture, such as clothing, toys, or eating 
utensils, which encourage hands-on learning about that culture. The 
concept of learning from objects is excellent. The choice of objects, how- 
ever, is crucial. 

We know of one object learning kit for elementary school children 
that includes the following objects on Japan; chopsticks, a folding fan, a 
summer kimono, origami paper with instructions, a wax replica of a piece 
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of sushi, and photos of children at a summer festival wearing kimono and 
playing games. The objects are colorful, fun to touch, and, to most U.S. 
children, unusual and intriguing. Unfortunately, the objects, taken to- 
gether, emphasize the exotic and the traditional. While all of these objects 
have their place in Japanese culture, without expert instruction in what 
these items represent, the overall picture ofjapan reinforces outdated 
images ofjapanese people wearing kimono, eating raw fish, and amusing 
themselves with origami. 

A second object learning kit on Japan includes the following objects 
on elementary education: a child’s backpack, a lunchbox, a calligraphy set, 
a math kit, a gym uniform, a school lunch menu, a newsletter from teacher 
to parents, and a report card. The kit includes descriptions for each item 
and translations for materials originally written in Japanese. This collection 
of objects does not try to depict all ofjapan, but rather focuses on one 
aspect of society, namely, elementary school. The objects are modern yet 
show the cultural trait of appreciating the traditional, as exemplified by the 
calligraphy set. The kit includes the perspectives of students and teachers 
and places each object into its proper context in Japanese culture. Because 
the kit focuses on elementary education, a topic all U.S. students know 
something about, the objects also encourage a comparison between school 
culture in the U.S. and that in Japan. 

The challenge for the evaluator, then, is to analyze the teacher’s 
choice of instructional materials and discover the reasons for his or her 
selection. This process will provide important insight into the teacher’s 
vision of culture and the impact of the cultural education program. The 
following sample questions can be used while examining the materials or 
during an interview with the teacher. While not complete, they may prove 
helpful in judging the cultural aspects of the teacher’s selection: 

a Are the facts correct? 

■ Is the information accurate and up-to-date? 

■ What features of culture come into view? What aspects are re- 
vealed? What is obscured? 
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B Does the material oversimplify culture to try to make it more 

''understandable? ' r ~ 

B Does it contain stereotypes or loaded words? 

B Does it make the culture seem exotic or inscrutable? 

H Does it excessively praise or criticize the culture? 

B Who wrote the material? 

B What point of view is included and what is excluded? 

a Docs the lesson incorporate culturally unfamiliar ways of learning 
or knowing? 

B How are opportunities for cultural reflection structured? 

B How are students encouraged to reflect on their own culture? 

B What does an object tell about who inade/owned it? What aspects 
of culture does it reveal? 

fl What variety and combinations of materials does the teacher use 
to present multiple perspectives within the target culture? 

In many cases the teacher will develop his or her own materials after 
participating in a cultural education program. Please see Chapter 5: Evalu- 
ating Instructional Materials for a more in-depth exploration of how to 
evaluate instructional materials. 

How does the teacher teach the content he or she has chosen? 

An effective cultural education program has the potential to change 
not only what a teacher teaches but how he or she teaches it. In our experi- 
ence with Japan-related programs, we have known teachers who have 
changed the way they teach about Japan, others who have modified the 
way they teach about all cultures, and still others who feel they have altered 
and expanded their repertoire of teaching methods. Through participation 
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in a cultural education program, a teacher may discover new teaching styles 
gleaned from both the target culture and the program. 

For example, a teacher may explore teaching styles found in the 
target culture as a way to teach about that culture. In the case ofjapan, a 
teacher may emphasize more group work in the classroom to emphasize 
the value of working harmoniously with others. For instance, we have 
known elementary school teachers who have adopted the Japanese educa- 
tional practice of using small fixed-membership groups ( han ) of four to six 
students to organize activities and accomplish tasks in the classroom 
(Quinn, 1995). One teacher who employed this method reported that she 
was removed from the center of attention in the classroom and was forced 
to relinquish some control to students. 

Teachers may also model their lessons after the cultural education 
programs in which they participate, using not only the same content or 
materials, but also the same method of teaching about another culture. As 
chapter 2 explained, a cultural encounter is a powerful learning tool, and a 
well-designed program provides participants with opportunities to interact 
face-to-face with another culture. Emulating their own experiences in the 
program, teachers may try to provide their students with similar opportu- 
nities for cultural encounters through guest speakers, films, objects, field 
trips, etc. This allows teachers to step aside as the mediator and allow 
students direct interaction with someone or something from another 
culture. 

For an evaluator, tools such as interviews, classroom observation, and 
an examination of lesson plans lend themselves to the discovery of changes 
in teaching style. In addition, the evaluator can also judge the teacher’s 
selection of guest speakers, field trips, films, etc. for evidence of his or her 
ability to construct sophisticated cultural encounters for students. 

How does the teacher evaluate student performance? 

When teachers’ goals for teaching about another culture change as a 
result of participating in a cultural education program, their evaluation of 
student performance might undergo a corresponding change. Before 
participating in a cultural education program, teachers might design a test 
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that asks students to display factual knowledge about the target culture, 
such as naming the four main islands of japan, or demonstrate skills, such 
as reading a map. After the program, however, teachers may wish to assess 
the degree to which their cultural goals have also been met. Teachers may 
develop creative new assessment tools to get at such questions as: Do 
students reveal more tolerance for people, places, and things that arc 
different? Do they display an understanding of how their own culture has 
shaped their values? Can they take the perspective of someone from 
another culture? 

To determine whether a cultural education program has had an 
impact on the way teachers evaluate student performance, the evaluator 
might examine tests and assignments, interview the teacher, and observe 
him or her in the classroom. To determine the effect of teachers’ new 
teaching behaviors in greater depth, the evaluator might interview students 
and examine their completed tests and assignments, including artwork, 
writing samples, speeches, etc. 



Knowledge 

It is obvious that after completing a cultural education program 
participants should know more than they did before about the concept of 
culture generally and about the culture under study specifically. The 
evaluator can compare participants’ knowledge of culture both pre- and 
post-program or can compare participating educators’ levels of knowledge 
with those of a non-participating control group. In either case, we recom- 
mend three kinds of data-gathering that provide evidence of knowledge: 
direct questioning, observation, and document analysis. 

First, a cultural education program evaluator can accomplish direct 
questioning through interviews, application essays, and questionnaires. 
Chapter 3: Evaluating Cultural Education Leadership discussed the impor- 
tance of program leaders condi cting in-depth interviews with participants 
at the start of a cultural education program. An evaluator may suggest 
questions to ask during this interview and, ideally, would be present when 
it occurs. The questions an evaluator might ask to gauge the sophistication 
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of participants’ knowledge of another culture will depend on the focus of 
the program but could include simple and open-ended questions, such as: 
“What are your impressions ofjapancsc women?” “What arc the major 
forms ofjapanese traditional theater?” or “How would you characterize 
Japan’s national security policy?” 

Initial interviews provide a wealth of information not only to the 
program leaders, but also to the evaluator and are a good starting point to 
establish a baseline of participants’ knowledge. If the evaluator is unable to 
be present during the interviews, then the program leader’s detailed notes 
or tape recordings of the conversations are essential. Exit interviews, 
conducted by the program leader and/or the evaluator at the conclusion of 
a program are equally revealing. Exit interview questions may be more 
nuanced and could include such questions as: “What role do you think 
culture plays in the trade imbalance between Japan and the United States?” 
“What have been the contributions ofjapanese artists to Western art?” or 
“How wo aid you characterize the teacher-student relationship in Japanese 
elementary schools?” 

If the program leaders and evaluators have the opportunity to consult 
with one another in advance of a program, then it might be possible for the 
evaluator to help design the program application form. Essay questions, 
such as “What value does the study of the Japanese education system hold 
for U.S. educators?” can prompt responses that offer insight into partici- 
pants’ preconceived notions of culture, the value of cultural education, or 
existing stereotypes about the target culture. Unfortunately, many program 
leaders do not hire an evaluator until after the application form is designed 
or after the participants are selected. If this is the case, evaluators can still 
review the forms completed by successful applicants. 

Questionnaires are probably the most common tool evaluators use to 
ask participants directly about their knowledge. Questionnaires adminis- 
tered before a program begins, daily instruments to be filled out as a 
program progresses, and surveys conducted at the completion of a pro- 
gram all provide useful information about what participants know and 
have learned. As an example, the following three questions were used by 
the evaluator and program leader at the completion of a graduate course at 
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the University of Maryland in 1993 that was part of an intensive year-long 
program that compared Japanese and U.S. education: 

1 . An array of education critics and scholars have recently suggested 
that the greatest cultural and education strengths of a nation might 
constitute its greatest weaknesses as well. Based on your studies of 
Japanese education and culture to date, do you agree with this ? 

2. There is a common assumption that Japanese teachers are auto- 
crats, securing conformity to rules and regulations, forcing stu- 
dents to memorize facts and figures, to parrot back, rote-like, the 
contents of the Course of Study, and otherwise master enormous 
quantities of information. Based on your studies of Japan so far, is 
this a reasonable characterization? If so, how? If not, how? 

3. From its beginnings in 1872, public education in japan has been 
an instrument of national government, its curricula the product of 
central planning, its teachers the mediators of its content, and its 
students the recipients of uniform content. In the U.S., on the 
other hand, public education has been controlled by state and local 
governments, its curricula varied, its teachers interpreters as well 
as mediators of its content, and its students the recipients of 
diverse fare. Based on your studies ofjapanese education tradi- 
tions, policies, and practices to date, please assess the utility of this 
characterization. 

t 

Another valuable way to assess participants’ levels of knowledge is to 
observe them in their workplaces. This is an excellent way to determine 
what elements of the program they have internalized and applied in their 
own individual educational settings. Many of the questions used to judge 
the cultural sophistication of instructional materials are also helpful in 
evaluating the knowledge level of participants. These questions can in- 
clude: Do teachers have their facts correct? Is their knowledge accurate and 
up-to-date? Does their knowledge go beyond stereotypes? Do they know a 
variety of perspectives from within the culture? Can they make nuanced 
comparisons across cultures? 
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We have observed teachers who returned from professional develop- 
ment seminars and conducted workshops for their colleagues in which ^ 
they accurately and sensitively passed on knowledge they gained about the 
Japanese education system. Unfortunately, we have also seen teachers who 
gave their students incorrect information after participating in a cultural 
education seminar. For example, we observed one teacher who wanted to 
illustrate greetings and forms of respect in japan and taught her students a 
bow that is common in southeast Asian countries but is not practiced in 
Japan. This is a significant error wben teaching about Japan, given the 
importance attached to the bow as a form of respect and a way to mark 
hierarchical relationships. The challenge for the evaluator is to determine 
whether this error can be traced to a poorly designed program or to insuf- 
ficient research/ study on the part of the participant. 

Various documents can also indicate a participant’s level of knowl- 
edge about the target culture. Journal articles, newspaper articles, grant 
proposals, or correspondence to parents and community groups are ex- 
amples of documents written by participants that may indicate a new level 
of knowledge sophistication. One rich source of evidence is the curricula 
that participants develop before and after a program. An excellent example 
we have encountered is a language arts unit on narratives of oppression 
developed by a graduate of a Japan-related seminar and her program leader 
that helps students explore discrimination, inequality, and injustice in the 
U.S. and Japan. Please see Chapter 5: Evaluating Instructional Materials for an 
in-depth look at this particular unit. 

In other cases, however, we have been dismayed when program 
participants have produced instructional materials that were shallow, 
contained inaccuracies, or promoted stereotypes instead of dispelling 
them. Sometimes a relatively minor flaw can lessen the impact of an 
otherwise creative lesson. Such was the case fora curriculum supervisor 
who returned from a Japan seminar and developed a unit about Japanese 
traditional clothing. She illustrated the unit using the figure of a Japanese 
woman wearing a kimono; however, she drew the kimono incorrectly 
(wrapped right over left instead ofleft over right around the body). Al- 
though it may seem minor, this is an important error because it demon- 
strates the participant’s lack of knowledge about the importance of form in 
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wearing a kimono. She had inadvertently drawn the figure the way a 
corpse is dressed for a funeral. r : “ ' 

Evaluators can also assess whether a cultural education program has 
an impact on a participant’s personal quest for further knowledge about the 
target culture. For example, after participating in one of our cultural 
education seminars in Japan, a college professor was inspired to enroll in 
his college’s undergraduate Japanese language course. Another alumnus of 
the same program continues to expand his collection of short stories and 
films from Japan, using sources he gained from the seminar. The section 
“Community Building” later in this chapter discusses other efforts to 
broaden knowledge about the target culture within communities. 



Communicative Disposition 

Cultural education programs that are rich in content emphasize the 
role of culture in human interactions. Evidence that participants are 
attuned to their own communicative dispositions indicates an ability to 
hear as well as listen, to attend to meaning as well as language, to adjust to 
different contexts, and to work with cultural and professional “others” in 
reciprocal and respectful ways. A sophisticated cultural education program 
encourages participants to “mute the tendency to (1) oversimplify in an 
attempt to make another culture more understandable, (2) overgeneralize 
to reduce feelings of ambiguity, and (3) become judgmental when encoun- 
ters with different cultural perspectives cause discomfort” (Finkelstein, 
Imamura, & Tobin, 1991, p. 3). 

Evaluators might index a greater sensitivity by attending to the way 
in which participants communicate in their own educational environ- 
ments, structure interactions, and otherwise approach teaching and learn- 
ing. Following a program in which participants have the opportunity to 
interact with people from other cultures and to reflect on those experi- 
ences, evaluators might ask whether participants are more comfortable in 
unfamiliar cultural situations. Evaluators might also compare the responses 
of participants and non-participants, or of participants pre- and post- 
program, to questions that ask what experience they have had in dealing 
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with people from other cultures and how they have dealt with cultural 
misunderstandings. For example, before a seminar fof teachers visiting 
Japan, an evaluator asked the following two questions: 

H Think back on a situation in which you were involved when a 
communication problem occurred because of differences in 
culture. What happened? Who was involved? How did you feel? 

■ Think of a situation in which you were involved where a potential 
communication problem involving cultural differences was 
avoided or resolved because of some action taken by you or some- 
one else. What happened? Who was involved? 

Through answers to questions like these, evaluators can determine 
the degree to which respondents are sensitive to the role of culture in 
communication and have reflected on their own responses to the dilem- 
mas that these encounters produce. 

Evaluators can also look for indications that participants are more 
aware of unspoken, yet culturally-laden, communication styles. Often, 
participants report being more conscious of similarities and differences in 
body language, personal space, or eye contact. Participants who delve 
deeper may discover the effects of culture in direct and indirect language, 
apparent assertiveness or passivity, what is spoken versus what is left un- 
said, or attention to social hierarchy. For example, in our experience 
designingjapan-related programs for U.S. educators, we have learned to 
emphasize the importance that many Japanese place on the ability to work 
harmoniously in a group. Because we include group as well as individual 
goals, many graduates of our programs report a greater awareness of the 
benefits and difficulties of working in groups, once they return to their 
own work places. 

Participants in cultural education programs may also come to a better 
understanding of how to promote harmony among diverse groups. For 
instance, after participating in such a program, a principal might use his or 
her school to promote friendly encounters among parents from culturally 
diverse groups who might otherwise not have an occasion to meet. 
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Teachers, too, have the opportunity every day in every classroomto 
organize social and cultural relations among their students. Every time 
they put students into groups, every time they arrange their desks, every 
time they engage them in questions and answers, they are influencing 
social and cultural relations among the children. When teachers assign 
students to groups, they determine who works with whom, whether the 
advanced learners will be separated from the average students, the boys 
from the girls, and the English speakers from the non-English speakers. 
Evaluators can explore whether teachers who have been through a cultural 
education program are aware of their reasons for and the impact of their 
grouping choices. In interviews and observations, evaluators can expect 
graduates of cultural education programs to demonstrate a sensitivity to 
cultural differences within their classrooms and, when possible, to give 
students opportunities to interact with people from diverse cultures. 

Evaluators can also pay attention to the communication patterns 
among and between the teacher and students. Much has been written 
about the effect of culture on communication styles (Philips, 1983; Lewis, 
1995; Estrin, 1993) so we will mention only a few questions here for 
evaluators to consider: 

■ Does the teacher provide a variety of ways for students to partici- 
pate in the classroom discourse? 

■ Does the teacher create opportunities for students to respond as a 
group rather than always requiring individual recitation? 

■ Does the teacher promote cooperative learning, encouraging 
students to work together in groups and evaluating them on their 
collaborative efforts? 

A teacher may also emulate cultural patterns found in the target 
culture as one way to teach about that culture. We have already mentioned 
the example in which teachers emphasize the value of working harmoni- 
ously with others by grouping students in han, a Japanese educational 
practice used to organize activities and accomplish tasks in the classroom. 
To facilitate the group work, the U.S. teachers in this case also changed 
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their classroom layout, grouping the children’s desks by han to encourage 
new forms of interaction. Another teacher has adopted the use of rotating 
toban (monitors) to encourage student leadership in daily classroom man- 
agement. Yet another has adapted the practice of morning greetings to 
encourage structure, respect, and reflection. 

To determine the extent to which participants attend to the effects of 
culture in their communication styles, evaluators might; 

■ question participants directly (With what cultures do you interact 
in your classroom/workplace/community? How would you char- 
acterize those interactions?); 

■ ask participants to keep journals of situations in which culture 
played a role (Who was involved? Was there a misunderstanding or 
miscornmunication? How was it resolved?); 

S analyze documents such as letters to parents, newsletters, or 
communication with a sister school in another country (When 
communicating with parents, what accommodations, if any, does 
the principal make to the culture of the family?); or 

■ observe participants in their educational settings (How does the 
teacher promote interactions between culturally diverse students, 
if at all?). 



Community Building 

Successful cultural education programs inspire and/or require educa- 
tors to move beyond personal growth and their own immediate classrooms 
and workplaces to reach out to a larger community and share what they 
have learned from their experiences. Participants may bring about changes 
m their schools, school districts, state departments of education, universi- 
ties, colleges, or communities in many ways. Because ' he outcomes can be 
diverse and the communities widespread, the challenge for evaluators is to 
discover the new behavior, determine whether it is an outcome of the 
program, and judge it for its cultural features. 
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Evaluators will find interviews, questionnaires, observation, and 
document analysis to be helpful tools. They may wish to extend their data 
collection beyond the immediate program participants to the communities 
they reach, for example to address the colleagues with whom participants 
consult, the attendees at the workshops they create, the beneficiaries of the 
funds they raise, or the participants in the exchanges they develop. 

In our experience with Japan-related programs, we have documented 
a wide variety of community-building outcomes. Some were immediate 
results, while others took years to come to fruition. Some represented 
small steps, while others demonstrated sweeping changes. Some were 
modest collaborations of individuals, while others involved entire school 
districts. Based on what we have observed, we have developed the follow- 
ing questions to help evaluators in their search. We do not mean to suggest 
that these are the only outcome possibilities, or that cultural education 
programs should produce results in all of these areas. Rather, because the 
ripple effect of programs can be extensive, these questions will help evalua- 
tors cast their nets wide in the pursuit of outcomes. In the remainder of 
tlais chapter, we discuss examples of each of the following questions: 

■ How does the participant serve as a resource on culture in general 
or the target culture in partis Tar? 

■ What new programs has the participant developed? 

■ What exchanges has the participant developed or facilitated? 

■ What funds does the participant raise? 

■ In what ways does the participant communicate and collaborate 
with other participants of the program? 

■ What professional relationships has the participant developed? 

Si To what networks/professional associations does the participant 
belong/contribute? 

■ At what professional meetings does the participant make presenta- 
tions? What is the content? 
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a What has the participant written/published professionally? Where 
was it published? What was the content? ' 

■ What research topics does the participant pursue? 

■ How is the participant involved in bringing about change in the 
community? 

How does the participant serve as a resource on culture in 
general or the target culture in particular? 

Following participation in a cultural education program, participants 
are often regarded as the “resident expert” on the particular culture ad- 
dressed in the program. It is not unusual for program graduates to be the 
only people in the workplace who have received in-depth training in a 
specific culture. Consequently, these participants are frequently called 
upon to clarify cultural points, recommend materials, suggest activities to 
enhance the established curriculum, or instruct others on the target cul- 
ture. They give much of this advice informally, such as when teachers 
discuss upcoming lessons or units. More formally, however, participants 
also serve on curriculum committees, present at workshops, select and 
purchase materials, or help school systems work with students and parents 
from the target culture. 

Evaluators can look at the cultural features of the workshops that 
participants conduct, the materials they purchase, the advice they give, and 
the interactions they promote for evidence of the quality of their cultural 
learning. Content-rich, culturally-nuanced education programs have the 
potential to produce graduates who become knowledgeable, sensitive, and 
culturally sophisticated resources for others. 

We have observed the following results after educators have partici- 
pated in japan-related cultural education programs: 

■ A kindergarten teacher traded classes with another teacher in the 
same building during the 4 japan unit” so that she shared her 
newly-acquired knowledge about Japan not only with her own 
class, but with the other kindergarten class in the school. 
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m An elementary school media center specialist learned about new 
resources, evaluated the holdings in her school’s media center, 
discovered what was lacking, and suggested new purchases to fill 
in the gaps. She now makes more knowledgeable recommenda- 
tions for the purchase of sophisticated materials. 

■ A principal in a school district that includes a unit on Japan in the 
third grade knew that many teachers felt inadequately prepared to 
teach the required lessons. She organized and conducted a work- 
shop on Japan for all third grade teachers in the district to increase 
their knowledge and sensitivity, build their confidence, and 
promote the use of new materials. 

■ A third grade teacher spoke with her colleagues about having daily 
school meetings to promote school spirit and feeling of commu- 
nity, a practice she observed in Japanese schools. Following this 
discussion, the principal changed the schedule in order to allow an 
all-school meeting in the gymnasium at the end of the day. 

What new programs has the participant developed? 

After participating in a cultural education seminar, program gradu- 
ates often envision new program possibilities that will enrich students, 
colleagues, and the community. These programs can result from collabora- 
tions developed during the seminar with other U.S. colleagues or with 
counterparts in the target culture. For instance, participants may create 
professional development programs, workshops, or language programs that 
reach new audiences; provide new opportunities; or are innovative, cultur- 
ally complex, or rich in new materials. 

The most ambitious program we have seen was a thirteen-year staff 
development initiative in a rural Maryland county. Initially, four county 
educators participated in a university-sponsored Japan-related seminar: a 
curriculum supervisor, a budget director, and two classroom teachers. As 
part of their team application for that seminar, they proposed a plan that 
would enable four county educators to study in Japan every year for the 
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next thirteen years: four kindergarten teachers the first year, four first 
grade teachers the second year, four second grade teachers the third year, 
etc. In this way, by the time the students who were then in kindergarten 
graduated in the year 200'', teachers from every grade would have received 
advanced training in Japan-related instruction. 

Ultimately, the plan did not unfold exactly as proposed. The county 
eventually opened the seminar to applicants from any grade and to non- 
teaching staff as well. Still, the program has been in existence for nine 
years, and thirty-six county educators have participated in summer semi- 
nars to Japan. The results have been extraordinary. The curriculum has 
been internationalized county-wide to include sophisticated Japan-related 
content. The program alumni have conducted workshops, created new 
language programs, developed instructional materials, created exchanges, 
and even written about the program in an education journal. The program 
is now institutionalized and has gained such broad support that it survived 
the change-over to a new superintendent and the departure of two of the 
original program developers. Having reaped the benefits of their work in 
one county, these two participants have moved to a neighboring school 
district and are sowing the seeds of new programs in new sites. 

Results from other cultural education programs include: 

■ A staff development specialist made a concerted effort while in 
Japan to collect objects that could be used for hands-on learning in 
the classroom. After she returned, she worked with a Japanese- 
American parent in the school system to organize the objects into 
“artifact trunks” around themes such as school life, food, clothing, 
etc. She developed a system through which any teacher in the 
school district could borrow the materials at no charge, a system 
she later expanded to include teachers in neighboring school 
districts. The materials in the kits are replenished periodically by 
the participants in the annual Japan seminar mentioned above. 

■ An elementary school teacher maintained her contacts with the 
staff of the university that coordinated the Japan seminar in which 
she participated and several years later worked with them to 
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coordinate a workshop on Japan for h er colleagues. Although it 
was originally intended for the teachers in her school, she ulti- 
mately gained the support of the assistant superintendent, and the 
workshop was expanded to include elementary teachers from 
across the district. In this way, the teacher was able to bring a new 
form of professional development to a broad segment of her 
colleagues, including new knowledge, teaching methods, and 
resources. She also observed the added benefit of workshop 
participants communicating and collaborating across schools and 
across grades. 

II A foreign language supervisor was instrumental in developing a 
Japanese language program at the middle and high school levels in 
her school district that had previously focused on European 
languages. With her support, the district has hired severaljapanese 
language teachers and has become a leader in the development of 
new curricula for teaching Japanese to middle school students. 

What exchanges has the participant developed or facilitated? 

After experiencing their own cultural encounters and living, work- 
ing, and playing with people from another culture, many educators are 
eager to develop exchanges that offer students and other colleagues similar 
experiences. Participants may seek to promote the exchange of people, 
ideas, or materials across cultures, including teacher exchanges, student 
exchanges, sister school relationships, E-mail projects, or pen pal pro- 
grams. We have also noted that alumni of cultural education programs are 
often willing host to someone from another country in their home or 
school as part of an exchange. Participants often cite a desire to “give 
something back” in return for the hospitality they received while in an- 
other country. 

The caliber of the exchanges that program graduates develop is a 
potential indicator of the quality of the original program. Evaluators can 
judge these exchanges using a variety of questions which might include: 
Does the exchange bring together culturally disparate groups in new forms 
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of collaboration? Does. each side treat the_culture.of.the other with respect 
and dignity? Does each side benefit equally from the exchange? Does the 
exchange live or die on the efforts of one or two individuals, or does it 
have a broad base of support to continue indefinitely? 

We have observed the following exchanges that have resulted from 
cultural education programs: 

S A. social studies supervisor in a state Department ofEducation has 
been asked to coordinate a Japanese Ministry ofEducation pro- 
gram that brings twenty Japanese teachers to her state every year to 
live in U.S. homes and work in U.S. schools for two months. The 
teachers come to learn about the U.S. and U.S. education and to 
teach about Japan and Japanese education. The social studies 
supervisor works with the Japanese Ministry ofEducation, the 
U.S. Department ofEducation, and local school districts to orga- 
nize school visits and homestays. The program is rich in opportu- 
nities for preparation and debriefing of the participants, including 
an orientation for the host families, a bilingual orientation for the 
Japanese teachers, and follow-up activities at the end of the pro- 
gram. This program has now operated for eight years with locali- 
ties across the state vying for the opportunity to participate. 

■ A curriculum supervisor worked with the university that orga- 
nized her Japan-seminar to establish an exchange between her 
school district and the Education Department of a small, private 
university in Japan. Every year for three weeks the school district 
hosts thirty pre-service teachers from the Japanese university. Each 
Japanese student is paired with a mentor U.S. educator, Lives in his 
or her home, observes in his or her school, and has the opportu- 
nity to student-teach in his or her classroom. In return, the Japa- 
nese university hosts four educators from the U.S. school district 
during an annual summer seminar in Japan that includes school 
visits, homestays, and cultural programs. 
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What funds does the participant raise? 

In order to support the programs and exchanges they initiate, many 
alumni of cultural education programs raise funds to support their projects, 
including fund-raising in the community and obtaining grants from state 
and federal departments of education, foundations, and nonprofit organi- 
zations. Evaluators might look for whether participants have the comm ; “ 
ment and initiative to seek new funds, whether they are successful in 
raising money, whether they use networks developed in the program to 
make contacts, and whether, through the program, they have learned 
about new sources of support such as cultural foundations, professional 
organizations, or international businesses. 

We have observed the following fund-raising activities by alumni of 
cultural education programs: 

H A curriculum supervisor received a grant from the U.S. Depart- 
ment of Education to initiate a partial immersion Japanese lan- 
guage program at an elementary school in her rural school district. 
The funding this grant provided was directly responsible for the 
hiring of a Japanese instructor. 

■ An elementary school teacher heard from the leader of her Japan 
seminar that another education organization awarded small grants 
for cultural projects. She applied, using the program leader as a 
reference, and was awarded a grant to travel to a national confer- 
ence to make a presentation on child-rearing practices in Japan. 

■ An elementary school teacher received a grant from a major 
newspaper to develop a video instructional unit that is currently in 
use in her school district. 

■ A budget director who participated in two intensive Japan semi- 
nars is committed to ensuring opportunities for others in his 
district to have the same kind of experience. From his position as 
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~ - finance director, he ensures that the school system’s budget in- 

eludes support for an annual seminar in Japan. Further, he solicits 
contributions from the community to supplement the district’s 
support. As a result, even through lean times, the Japan seminar 
has continued uninterrupted for nine years. 

In what ways does the participant communicate and collaborate 
with other participants of the program? 

One result of cultural education programs may be that participants 
find new ways to collaborate with one another across grades, disciplines, 
schools, and school districts. Participants may work together on such 
projects as curriculum units, articles, exchanges, presentations, team 
teaching, or research. Evaluators can look for evidence that alumni wish to 
continue their association, privately or through newsletters, regular gather- 
ings, or electronic communication such as bulletin boards. Evaluators may 
also wish to assess the quality of the collaborations into which former 
participants enter for their cultural sensitivity and sophistication. Finally, 
evaluators can look for new collaborations between previously discon- 
nected groups of educators. As program designers and evaluators, we have 
been especially gratified to witness prccollege educators and university 
professors working together, a relationship that can often be strained by 
status and culture distinctions. 

We have observed the following collaborations between graduates of 
cultural education programs: 

H Two alumni from different years of an intensive Japan seminar, a 
university professor and a curriculum supervisor, co-authored an 
article that was published in a leading educational journal on the 
benefits of a long-term Japan-related professional development- 
initiative. 

■ A university professor and a high school teacher coordinated an 
exchange of ideas between their students through electronic mail. 
The university students, all of whom were pre-service teachers, 
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:: . -i -bcncfitted froiH their interaction with high school students, .while ~ 
the high school students enjoyed discussing topics of mutual 
interest with older students, 

■ Two participants in a Japan-related arts seminar — an English 
teacher and a drama teacher — work with the art teacher in their 
high school every year to create a unique, multi-disciplinary 
opportunity for their students to learn about Japan. The students 
in the English, drama, and art classes learn about the Japanese tea 
ceremony through required readings, a writing assignment, and a 
video. Then the students participate in a tea ceremony performed 
at the school by a member of the local Japanese community. 
Finally, the students of all three classes move to the art studio 
where they create their own tea bowls using the aesthetic prin- 
ciples learned. 

What professional relationships has the participant developed? 

In addition to new relationships with program leaders, follow partici- 
pants, and other colleagues in the U.S., many participants develop or 
deepen professional relationships outside the country that were originally 
formed during a cultural education program. For some participants, these 
contacts open a new world of international collaboration and cooperation. 
We have known participants who have co-authored articles with Japanese 
counterparts, conducted joint research projects with Japanese colleagues, 
or invited Japanese associates to study in the U.S. As with relationships 
between program participants, evaluators can inquire about the breadth 
and depth of the associations, determine if they bring together formerly 
isolated groups, and assess the cultural features of any collaborations that 
develop. We have witnessed the following outcomes: 

■ A U.S. professor of education who had previously only communi- 
cated with a Japanese professor in letters was able in her Japan 
seminar to meet him face-to-face. She attended a meeting of the 
intercultural education society of which he was president, and he 
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introduced her to other Japanese in the field. Since that time, the 
U.S. professor has hosted the Japanese professor on a research trip 
to explore the policies and practices of education for foreign 
children in U.S. schools. The U.S. professor, in turn, introduced 
her Japanese colleague to other U.S. leaders of intercultural educa- 
tion. Later, the Japanese professor invited the U.S. professor to 
Japan to make a presentation at a conference on intercultural educa- 
tion, ensuring a continuation and deepening of the relationship. 

ffi An intercultural educator in the U.S. was introduced to ajapanese 
professor during her seminar in Japan. Later, both became doctoral 
students at a U.S. university where they studied together and 
collaborated on projects and papers. Based on his respect for her 
professional capabilities, the Japanese professor’s university offered 
the U.S. educator a teaching position in intercultural communica- 
tion and American studies. 

To what networks/professional associations does the 
participant belong/contribute? 

A cultural education program may have an impact on a participant by 
introducing him or her to new people and networks. Evaluators can look 
for evidence that, based on interests or contacts developed through the 
program, a participant has expanded his or her networks, joined a new 
network or professional association, or attended different professional 
meetings. These connections may lead to an entirely new array of possi- 
bilities for collaboration or the exchange of ideas. We have observed the 
following example: 

■ As an education historian, a university professor maintained an 
extensive network of colleagues, attended professional meetings in 
her field, and collaborated with other historians on projects and 
publications. Once she became involved in Japan-related pro- 
grams, however, she expanded her networks to include Japan 
experts — both Americans andjapanese — in such fields as educa- 
tion, government, the arts, politics, economics, ana language. She 
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began to attend annual meetings of Japan-related professional 
organizations, such as the Association for Asian "Si 4es and the 
Committee for Teaching about Asia. Through these new net- 
works, in addition to ongoing collaborations with education 
historians, she also works with Japan-related scholars and educa- 
tion professionals on books, articles, and research projects. 

At what professional meetings does the participant make 
presentations? What is the content? 

A cultural education program may also influence participants to 
speak at professional meetings in order to share their augmented knowl- 
edge and views. We have known participants who have gained such confi- 
dence in their mastery of the cultural content and in themselves that they 
were inspired to make their first presentation ever at a conference. In 
addition to documenting the existence of new presentations and assessing 
the forums participants choose, evaluators may also wish to examine the 
content of the presentations for evidence that the program has influenced 
the substance or heightened its cultural sensitivity. Some examples we have 
observed include: 

■ At a conference on elementary education, a kindergarten teacher 
gave a presentation on Japanese traditional storytelling ( kamishibai ), 
a topic she learned about in two Japan seminars. This was her first 
presentation at a national conference and marked a milestone in 
her professional career. 

■ Two high school alumni of Japan-re la ted programs worked with 
one of the leaders of their Japan seminar to develop a presentation 
for a national conference of English teachers. All were master 
teachers and the presentation — on the topic of introducing Japan 
into the study of the humanities — represented years of work by all 
three on how to integrate the unfamiliar, that is, the study of 
another culture, into familiar English topics such as poetry, drama, 
and literature. 
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What has the participant written/published professionally? 
Where was it published? What was the content? 

Participants, particularly university faculty and administrators, may 
be motivated by experiences in a cultural education program to write and 
publish professionally. What participants wrote before their experience and 
what they write following the program can be compared for changes in 
content and cultural sensitivity. An evaluator might also judge the prestige 
of the publisher or the participant’s writing productivity for further poten- 
tial evidence of the impact of a program. Examples include: 

H A curriculum supervisor and a professor of education, both alumni 
of the same Japan seminar, co-authored an article describing a 
multi-year Japan-related staff development program in a rural 
Maryland school district. The article was published in a respected 
education journal in which neither author had published before. 

■ A museum educator used information and contacts from her Japan 
seminar to write a curriculum unit on using objects to teach about 
another culture. The unit will be published together with other 
intercultural units by a leading educational press. This is the 
educator’s first piece published by a major press. 

What research topics does the participant pursue? 

After participating in a cultural education seminar, participants may 
engage in new types of research, delve into new research topics, or enter 
into research collaborations with U.S. or international colleagues. Evalua- 
tors can examine any of these for evidence of new vision, knowledge, or 
communicative disposition. We have observed these examples: 

■ Through Japan-related programs, a U.S. professor of education 
became intrigued with how the Japanese education system deds 
with children outside the mainstream, such as foreign children, 
minority children, and Japanese children who have received part 
of their education abroad. She began a collaboration with a profes- 
sor in Japan to conduct comparative research on the education of 
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foreign children in public schools. These professors hope that the 
results of their research will lead to more enlightened education 
policies and practices in both countries, 

■ A U.S. intercultural educator who both leads Japan-related initia- 
tives and participates in advanced Japan seminars credits one 
seminar with introducing her to the study of cultural and ethnic 
minorities in Japan, such as Chinese, Koreans, and burakumin 
(descendants of a former outcast class). In addition to her own 
studies, she now includes the topic when she creates cultural 
education programs for other U.S. educators who, she has found, 
often do not realize that minorities exist in “homogeneous” Japan. 

How is the participant involved in bringing about change 
in the community? 

Finally, following participation in a cultural education seminar, 
participants may share the results of their experience in non-educational 
settings as well as in their workplaces. In our experience, the activities in 
which they engage are as diverse as the participants. For instance, former 
participants might speak to a philanthropic organization, promote sister 
city or sister state relationships, organize an exhibit or event at a local fair, 
host an exchange student, or write for a newspaper or magazine. Evalua- 
tors can look for cultural expressions of participants’ vision, knowledge, or 
communicative disposition in many domains. These expressions may 
appear as an understanding of how to portray another culture sensitively 
without overgeneralizing or falling back on stereotypes, knowledge of 
materials and resources that can make another culture come alive for a 
general audience, the ability to interact effectively with people from other 
cultures, or sensitivity when dealing with various cultures within the 
community. Results we have observed include: 

■ Following her participation in ajapan seminar, an elementary 
school teacher spoke to the members of her church on her experi- 
ences attending Christian churches in Japan, emphasizing the 
similarities in faith that she felt so strongly. 
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JS An elementary school teacher chronicled her experiences in Japan 
in"a series of articles in a local newspaper that she hoped would 
broaden the cultural horizons of her community. 



Conclusion 

This chapter addresses the need for evaluators to document the many 
changes in behavior that can result from effective cultural education 
programs. Cultural vision, knowledge, communicative disposition, and 
community building are useful categories for assessing the qualities of 
successful participants. With these concepts, educators can begin to re- 
think what they truly want participants to gain from cultural education 
experiences, and evaluators can document the substantive changes in 
approaches to cultural teaching and learning that result from participation 
in a program. We hope that the examples in this chapter provide a frame- 
work for detecting these kinds of changes. By capturing a broad range of 
outcomes, evaluators can judge the worth of particular cultural education 
programs and their potential for improving teaching about other cultures 
and helping schools to guide their students to live successfully in a 
multicultural world. 
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T 

i^Lhe effects of sophisticated cultural education 
programs are no more graphically revealed or ame- 
nable to evaluation than they are in the quality of 
instructional materials that become available for use in classrooms, staff 
development programs, and curriculum transformation initiatives. When 
educators prepare instructional materials, they play important roles as 
cultural message carriers — presenting images of foreign peoples and cul- 
tures, shaping reactions to strangers, defining boundaries among and 
between students, and cultivating a sense of place . 1 

In evaluating lessons involving culture, evaluators need to identify 
the concept of culture embedded in the lesson's objectives, the choice of 
sources, the lesson procedures, and student evaluations. Evaluators must 
attend to the cultural sensitivity of the instructional materials as well as 
their factual accuracy, their implicit cultural messages as well as their stated 
aims, their relative sophistication, and their value for cultural conscious- 




1 The concept of curriculum policies as message forms and teachers as message conveyers has been developed by 
an array of scholars. For some examples see: Bernstein, B. (1914). Social class, language and socialization; 
and Class and pedagogies: Visible and invisible . In J. Karabel &A.H. Halsey, Ideology and power in 
education (pp. 511-534). New York: Oxford University Press . See also: Finkelstein , B. (1989). 
Governing the young: Teacher behavior in popular primary schools in 19th century United 
States. London: The Falmer Press, and Finkelstein, B. (1991). Dollars and dreams: Classrooms as 
fictitious message systems, 1790-1930. History of Education Quarterly, 31(4), 463-481. 
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ness-raising as well as for being cognitively stimulating. For example, 
evaluators can ask the following kinds of questions: 

1 . What concept of culture underlies the lesson? 

2. Is the information about the culture accurate?. 

3. What features of culture(s) come into view? 

4. Does the lesson incorporate culturally unfamiliar ways of learning 
or knowing? 

5. How are opportunities for cultural reflection structured? 

6. How are learners encouraged to reflect on their own cultures? 

Evaluators can also look at how instructional materials help learners 
develop an introspective stance that allows them to examine cultural 
influences on their own behavior. 

In this chapter, we describe two different examples of instructional 
materials and discuss how to evaluate their cultural features. The first case 
is a set of lessons that invites students to create a series of haiku poems as a 
form of cultural expression. Haiku is a type of poetry requiring a specified 
syllabic length and references to seasons or other aspects of the natural 
world. It emerged in 19th century Japan from an earlier form of linked 
poetry known as renga. Renga were created by several authors who wrote 
individual haiku first and then linked them together with connecting 
couplets. The first case addressed in this chapter invites students to investi- 
gate culturally shaped images of nature as well as cultural attitudes toward 
poetry. 

The second case illustrates how narratives can help students examine 
culturally uncomfortable subjects, such as discrimination, in a legitimate 
classroom context. Discrimination is present in all cultures and in many 
varieties of literature, but especially biographies and autobiographies reveal 
this darker side of humanity. These instructional materials reveal the hurt, 
grief, rage, despair, and emotional costs to members of a minority group in 
Japan known as burakumin. 
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Both cases are written for high school language arts classes. Both link 
lesson content to reading and writing improvement as well as to cultural 
exploration. Each case aims to integrate cultural issues into the study of 
literatures, poetic expression, and history, but the two cases, as we shall 
see, carry different cultural messages. The evaluator's task is to capture 
what aspects of culture the lessons portray and how well they do this. Dr, 
Nancy Traubitz, a high school English teacher and curriculum specialist 
from Springbrook High School in Montgomery County, Maryland is the 
primary author of both sets of instructional materials. Dr. Traubitz worked 
with two of this book’s authors to expand the cultural dimensions of the 
lessons in different directions: the poetry lessons with Pickert; the narra- 
tive lessons with Finkelstein. First we describe the cases, and then we 
evaluate them. 




Linking Haiku: A Japanese Renga Party 
for American Classrooms 

Introduction 

The first set of lessons, designed as part of a high school English 
curriculum, discusses cultural metaphors embedded in several haiku that 
are translated from Japanese and also gives students specific information 
about the technical aspects of interpreting and writing haiku. While haiku 
poems appear to be only a brief description of a specific event without a 
plot, they are actually governed by elaborate principles that can take years 
to master. Nevertheless, most people agree on general rules for haiku. 
Henderson (1967) suggests four general guidelines for the study of haiku. 
Each poem.: 

■ consists of 17 Japanese syllables (5-7-5); 

■ contains some reference to nature (other than human nature); 
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■ refers to a particular event (i.e,, it is not a generalization); and 
. . Si presents that event as happening now— not in the past.; 

Linked poetry, or renga, has a history in Japan that stretches over 
several centuries, from 1200-1800 A.D. The practice of creating linked 
poems as a cooperative effort among authors flourished during uncertain 
and dangerous times in japan’s history when military rulers were deeply 
influenced by the philosophy of Zen Buddhism. The Zen emphasis on the 
fleeting impermanence of life shaped the principles that govern this an- 
cient haiku form. Today, there are many schools of haiku in Japan and 
other countries, each with its own rules. One objective of the haiku 
lessons is to help students explore cultural differences of ancient and 
modern Japanese literary traditions as well as cultural distinctions that are 
evident in English andjapanese versions of haiku. 

The following lessons are designed to be carried out as small group 
collaborative reading and writing projects, reflecting the way this same 
activity might be conducted in Japan today. They require three 50-minute 
class periods. The first period introduces students to the technical aspects 
of haiku and couplet writing. Students discuss Japanese rules for writing 
haiku and ways to identify seasonal motifs. In the second class period, 
students reassemble 16 haiku that have been separated by line. Then they 
compose linking couplets between the haiku and sequence the poems 
according to seasons of the year. This long linked haiku makes a renga. 
Finally, students read their completed renga in a class poetry party. In the 
third class period, students reflect on the poetry party experience. 

Students apply the haiku convention of identifying different seasons 
of the year by using metaphors from nature — flowers, animals, and in- 
sects — that must appear in every Japanese haiku, to explore how different 
cultures conceptualize and represent nature in language. The lessons ask 
students to judge a “successful” renga according to these Japanese metaphor 
rules and to determine what criteria they would use to evaluate theii own 
renga . Finally, students reflect on the ways culture shapes social interactions 
by critiquing the collaborative approach to poetry lessons. 
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Goals of the Lesson 

■ To recognize .universal.. themes valued: in all great literature as well 
as distinct cultural differences that are magnified by the concrete 
and specific imagery required by this poetic form. 

■ To gain a better understanding of how culture influences choice 
of words and forms of interaction among people. 

In this collaborative lesson, students study and practice using one 
poetic form (haiku) while simultaneously learning a little about how 
poetry reflects the culture in which it was created. Students use poetry 
written in Japan many years ago as models for verses that reflect their own 
contemporary perspectives to create a valuable cultural experience. 

Materials 

Materials include one “frog” haiku to introduce the unit (included 
here) and sixteen haiku translated from Japanese to English for the class 
activity. (For the full unit, please see Traubitz, 1995 in the Teacher Re- 
sources section below.) Recommended selections come from Haiku in 
English, translated by Harold G. Henderson (1967). The first set of four 
haiku are by early 17th century authors whose identities are unknown. 
The second set are by Basho (1644-1694). The third set are by Buson 
(1715-1783). The fourth set a-e by Issa (1762-1826). These famous Japa- 
nese authors have distinct styles, and their poems can be used to highlight 
individual variations amongjapanese poets. 



Lesson 1: Understanding Haiku and Renga 

Objective: To read several translations of a single poem and consider 
the influence of language structure and cultural meanings of particular 
images in poetry. 

Procedure: During the introductory lesson, ask the class to learn a 
few rules for haiku. Discuss the difficulties translators have in capturing 
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the core idea and spirit of haiku while attempting to follow the traditional 
17 syllable length restriction. Translating these poems from Japanese into 
English creates special challenges because Japanese is a syllabic language 
with most words ending in a vowel, while meter and syllable stress are 
more important in measuring the length of an utterance in English. These 
differences, coupled with difficulties in translating metaphors across 
cultures and languages, make it hard to stay within the 17 syllable require- 
ment when constructing English versions of haiku. In addition, Japanese 
haiku require special punctuation conventions that have no direct English 
equivalent. Teachers can use the sample “frog” haiku, to be described, to 
bring out these points before beginning Lesson #2. 

The “frog” haiku is the most popular haiku ever written. Dozens of 
different translations make it a good example for teaching about translation 
options and about how culture influences images of nature. Teachers 
should point out the importance of simplicity and formal structure re- 
quired by the haiku form and ask students to consider translation options 
that would satisfy both these requirements. 

Begin the lesson by asking students to choose their favorite transla- 
tion of the poem below and defend their choice. 



Basho’s poem 
fttruike ya 
kawazu tobikomu 
miztt no oto 

Translation #i: 
old pond . . . 
a frog leaps in 
water’s sound 

(translated by Higginson & 
Harter, 1989) 



(literal translation) 

(old pond . . .) 

(frog leap into) 

(water’s sound) 

Translation it 2 : 

Listen! a frog 
Jumping into the stillness 
Of an ancient pond! 

(translated by Henderson, 1958) 



Note that in Translation #2, Henderson chose meaning over form, 
preferring to retain the spirit and visual image of the haiku. In doing so, he 
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violated certain formal rules that apply to the structure ofjapanese, but not 
English. Thus, the lilies do not always match the 5-7-5 syllable formula. 

Images like the ones in the “frog” haiku capture concrete moments 
but leave the meaning abstract and open to interpretation. Basho sought to 
create a sense of quiet in his poems. Docs the frog’s 'splash intensify the 
pond’s stillness? 

Teachers can extend the lesson to other kinds ofjapanese art where 
nature themes arc prominent, for example, in paintings, scrolls, screens, 
and designs on silk. Hundreds of animals, insects, and plants are metaphors 
for particular seasons of the year. In this haiku, the frog represents spring. 



Lesson 2: Constructing Renga 
and Having a Poetry Party 

Objectives: 

■ To use seasonal motifs in haiku to identify the four seasons in 
selected poems. 

■ To reassemble Japanese haiku. 

■ To practice writing syllabic couplets to link haiku together, 
creating a renga. 

■ To participate in a poetry party like those popular in Japan in the 
16th and 17th centuries. 

Procedure: You can divide up the poems differently depending 
upon the size of your class. Make two copies of each poem and always keep 
one copy intact to serve as a reference and answer key. Poems do get 
scrambled as students begin to work with individual lines. Use different 
colored paper and/or different type fonts to clearly distinguish poems, 
authors, and seasons from one another. If you give each student one line 
and use all the lines of all four sets, you can work with 48 students. For 
these large classes, cut apart each poem and mount each line on a 4 x 6 



1 3 b 



134 




Chapter 5: Evaluating Instructional Materials 



inch card. If you code each line as la, 2a, or 3a on the back of the card, you 
will ensure you can eventually match up the right lines, in the right order, 
for the right poem as you reassemble the haiku. Note that interesting 
poems can also result even if the lines are sequenced incorrectly. 

You can also handle a group of 16 students by assigning each person a 
complete poem. If you assign two students to each poem, you can use the 
materials with 32 students. If you divide the poems by authors or seasons, 
you can work with a group of four students. Removing all the poems by 
one author from the set allows a smaller number of players to play while 
keeping all four seasons and still permitting comparisons among authors. 

When working with individual lines, first ask students to find ail 
three lines that make up their poem. When each poem has been reas- 
sembled, ask the students to identify the season described in the poem. 
Then divide students into four groups, with each group representing all 
four seasons of the year. Students should first order the poems within their 
groups starting with the current season. Each group should then compose 
three seven-syllable couplets linking the four poems within the set and two 
seven-syllable couplets linking the set to two other poetry groups. Each 
group's product should then be five seven-syllable couplets placed in the 
following way: 7-haiku (spring)-7-7-haiku (summer)-7-7~haiku (fall)— 7— 
7-haiku (winter)-7. Finally the class comes together, joining all the haiku 
into one renga made up of four years with four seasons each. An alternative 
arrangement is to assign all students with poems representing the same 
season to the same group. Each group then writes the linking couplets and 
the class product becomes a renga representing one year with four haiku 
about each season. 

Before beginning the class poetry reading, describe how poetry 
parties were a form of entertainment in Japan’s early history and how it 
was important for all nobility to be able to compose poetry. The references 
to follow provide information on these parties. When each group has 
completed its links, ask the class to enjoy its own renga party by reading the 
entire set of poems with the added linking couplets. This reading should 
begin with the current season. 
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Lesson 3: Debriefing Session- 
Poetry and Culture 

Objective: To discuss why it is sometimes difficult to interpret 
poetry written by a person from another culture. 

Procedure: In the class period following the poetry party, ask the . 
students to evaluate their products based on standards set by the class and 
on Japanese standards. They should discuss their reactions to the experi- 
ence, their views on working in groups, and their attempts to create linked 
verse for Japanese poems. The following questions may prompt an interest- 
ing discussion: 

■ Why do you think it was so hard to classify the poems by the 
season of the year if all the poems were required to have “season 
words” in them? (Not all cultures use the same symbols for sea- 
sons, e.g., persimmon does not necessarily mean the fall season in 
the U.S.) 

■ Did the words you used in your linking verses suggest ideas that 
someone from another culture might find difficult to understand? 
(Example: fireworks suggest a summer Fourth ofjuly indepen- 
dence celebration; “Ho, Ho, Ho” might refer to the winter 
Christmas season.) 

■ Are there any puns or other humor in your poetry? Would some- 
one from another culture understand them same way you meant 
them? Flow would you explain the humor? Do you think humor 
is “culture bound?” 

SB Why are haiku sc short? What can you really express in so few 
words? Does your culture value someone who can express himself 
or herself concisely? Is “a person of few words” a compliment or a 
criticism? Which kind of person do you think is valued in Japan? 
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_ H How did you feel about all the xules that Japanese haiku requires? 
(Expect students to say: confined, constricted, frustrated. Tradi- 
tional Japanese arts frequently require that creative activities follow 
specific forms. Instruction often stresses lengthy memorization 
and practice of specific skills, activities that are not always valued 
highly in the U.S.) 

■ How did you feel about working in a group for this project? Were 
you concerned or angry that others wouldn’t know about your 
contribution? Did you think you didn’t need to work hard be- 
cause no one would know what you actually wrote? Were you 
happy/unhappy with how your partners worked with you? Is there 
anything in your educational experiences that might explain these 
reactions? 

■ What poems would you send to Japan to help high school students 
learn about the U.S.? What does your selection say about what 
you think is important about where you live? 

Encourage students to write in a journal throughout this set of 
lessons. The following questions are prompts for reflective writing activities: 

■ Describe a situation in which you used words that others did not 
understand. What did you do? 

■ Formal rules for writing haiku and renga are important in Japanese 
culture. What formal rules about writing are important in the U.S.? 

■ What did you think about creating a poem as a class project and 
being given a group grade? 

■ Being able to write poetry is still a valued skill in Japan because it 
encourages the writer to be concise, disciplined, and focused. 
What skills do you think help develop these qualities in a person? 
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Optional Activities 

■ Videotape or tape record the final class reading. Invite other classes 
to watch. 

■ Encourage students to bring in nature poems from other countries. 

■ Create an interdisciplinary exercise by providing links to other 
subjects. Nature poems can lead to discussions of ecology and the 
environment and to depictions of nature in art and music. Poems 
written in other countries during the sr.me historical periods can 
provide additional comparative perspe ives. 

Student Evaluation 

The poetry party’s success depends upon the cooperation of all 
players. Each group receives credit for identifying the seasons represented 
in its poems, for completing a section of the total poem in a way that 
follows established rules, and for participating in the final poem recitation. 
Students can be evaluated informally for their participation as group 
members, for their comments during the third class period, and for their 
awareness of how culture influences their own lives. 

Teacher Resources 

Britton, D. (Trans,). (1989). A haiku jturney: Basho’s narrow road to afar province . New York: 
Kodansha International. 

Henderson, H. G. (Trans.). (1958). An introduction to haiku: An anthology of poems and poets from 
Basho to Shiki. New York: Doubleday Anchor. 

Henderson, H. G. (1967). Haiku in English. Rutland, VT: Tuttle. 

Higginson, W. J. (with Harter, R). (1989). The haiku handbook: How to write t share , and teach 
haiku . New York: Kodansha International. 

Sato, H. (1983). One hundred frogs: from renga to haiku in English . New York: WcatherhilL (A 
collection of translations and illustrations of the famous “frog” haiku.) 

Stanford University Program on International and Cross-Cultural Education (SPICE). (1989). 
The haiku moment: Seeing the world in a grain of sand. Stanford, CA: Stanford University. 
(This packet contains haiku, including the “frog” haiku, slides, Japanese music, and 
suggestions for grade 7-12 writing activities.) 
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Trauhitz, N. (1995), Linking haiku: A Japanese renga party for American classrooms. College Park, 
MD: International Center for the Study of Education Policy and Human Valuesr(For 
more information about this unit, please write to the International Center at: Depart- 
ment of Education Policy, Planning, and Administration, University of Maryland, 
College Park, Maryland 20742.) 

Case Evaluation 

We pose six questions to evaluate the cultural features of these 
instructional materials. They are derived from the evaluation criteria 
developed for cultural education program materials discussed in chapter 2. 

1. What concept of culture underlies the lesson? 

This set of lessons suggests several concepts of culture that 
permeate the instructional goals, activities, and student evaluation 
criteria. These lessons make the following assumptions about culture: 

■ Culture is a force — it shapes people’s behavior, thinking, 
works of art, and especially language. 

■ We can learn about culture through studying language and 
through metaphors and artistic products like works of 
literature. 

■ We can also learn about culture through face-to-face inter- 
actions like cooperative writing activities and poetry recita- 
tion parties. 

■ All cultures have rules, and people make judgments about 
cultural products based on these rules. 

H Concepts of nature and of beauty differ across cultures. 

■ Common stories or themes appear in all cultures. 

■ Cultures change with time. In this case, the culture of Japan 
was different in the 1500s than it is today. 

■ Analyzing cultural products is one way to understand 
another culture. 
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2. Is the information about the culture accurate? 

The lessons provide considerable background about the role of 
poetry in Japanese culture. The instructional materials include 
accurate information and references that can lead to more discussions 
about the topic. Several English versions of a single Japanese poem 
show the difficulties of providing accurate translations. 

3. What features of culture(s) come into view? 

The lessons introduce students to new ways of evaluating 
cultural products. Questions for class discussions encourage students 
to contemplate features of culture along the following dimensions: 

(a) common and particular metaphors for seasons of the year; (b) 
formal and informal approaches to writing haiku; and (c) group- 
constructed versus individually-constructed poems. These haiku 
lessons encourage students to explore cultural attitudes concerning 
the quality of an artistic product. 

4. Does the lesson incorporate culturally unfamiliar ways of 
learning or knowing? 

The cooperative group approach to this activity mirrors a 
common approach to work assignments in Japan. Student evaluation 
criteria are consonant with the importance placed on cooperative 
group efforts. While cooperative learning is not unfamiliar in U.S. 
schools, it is not as common as other methods of instruction, particu- 
larly at the secondary level. 

5. How are opportunities for cultural reflection structured? 

The lessons incorporate both public (class discussions) and 
private (journal writing) opportunities for structuring reflective 
activities. Open-ended questions permit multiple interpretations of 
the topics and the experience. 

6. How are learners encouraged to reflect on their own 
cultures? 

These lessons suggest that the study of poetry can show stu- 
dents that they are cultural participants in their own classrooms, 
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negotiating rules and proclaiming values about standards for language 
use and interpersonal behavior. One assignment requires students to 
evaluate several translations of the same poem where they confront 
linguistic differences in nature metaphors and in rules for writing 
haiku. The discussion questions and the journal prompts also en- 
courage students to use their own personal experiences to draw 
meaning from a culture that differs from their own. 

The approach to poetry from a comparative perspective encourages 
students to think about the assumptions they make about people and 
nature in their everyday language and the resulting possibilities for misun- 
derstanding and disagreement among classmates who may not code expe- 
riences in the same way. 

These poetry lessons constitute a way to engage a group of students 
in an examination of how culture influences choice of words and attitudes 
about artistic products. They assume that cultures are coded in language, 
poems are cultural products, and that studying poems from another culture 
will help students recognize cultural biases in their own lai uage use. 



CASE #2: 



Narratives as Expressions of Culture 

Introduction 

Few types of literature illuminate culture as poignantly or as provoca- 
tively as personal narratives. Through the prism of personal recollection 
and lived experience, writers offer powerful lenses through which people 
can locate their own experience and communicate and share it with 
others. These stories sometimes raise unpleasant topics that are difficult to 
discuss, such as oppression or discrimination. Lessons that incorporate 
carefully-selected personal narratives potentially provide an avenue for 
students to examine taboo words and cultural values within a classroom 
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context. Through the interpretive framework ol their creators, the per- 
sonal narratives illustrate the fact that while human cultures have a dark 
side, human beings are also resilient and strong. The following two lesson 
summaries illustrate their creators’ efforts to introduce stories about 
discrimination to high school English students. Eleven complete lessons 
comprising a reading and a writing unit can be found in Finkelstein and 
Eder’s Messages of Culture: Instructional Materials for Critical Cultural Reflection 
(tentative title, in press). 

The authors of this lesson believe that narratives of discrimination, 
when carefully studied and sensitively presented, invite both teachers and 
students to approach painful, unrecognized, unfortunate, and unattended 
tendencies in every culture: to label and degrade those who are different; 
to deny the presence of discrimination, inequality, underprivilege, and 
injustice; to side with those who are powerful; to blame those who are 
weak; and to pick on the helpless. Narratives of discrimination often 
reveal the power of language to sustain conflict as well as to cultivate 
understanding. 

This study of narratives about prejudice is structured to provide both 
the means to understand how discrimination is sustained and reproduced, 
and a way to overcome its more virulent forms. One of the goals of the 
lessons is to introduce students to new voices and to help them discover 
new elements of culture. Because of the emotionally sensitive nature of 
this subject matter, students work in small groups where they should feel 
less embarrassed about discussing their own experiences with prejudice. 

To come face-to-face with this dark side of humanity in the company of 
others invites students to feel less isolated, to develop sympathy and under- 
standing for others, and to see themselves more clearly. As students study 
these narratives, they can also connect their own experiences with dis- 
crimination with those of the writers and their subjects. 

Goab of the Lesson 

■ To identify prejudice. To help students conceptualize the specific 
forms that prejudice takes in the narrative. To name which people 
make up the privileged and outcast groups in the readings. 
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a To study language as an instrument of prejudice. To identify 
techniques used to demean and insult. To determine what weap- 
ons the in-group uses against the out-group. 

B To analyze personal narratives that communicate experiences of 
prejudice. To consider what narrative techniques the author uses 
to communicate the experience of prejudice, what perspectives 
the author reveals about the cultural values of the privileged in- 
group and out-group, and how the prose communicates the 
experience of discrimination. 

B To write about and share a personal experience with prejudice. 



Time 

One classroom period for each lesson. 

Materials 

List of “bad words” from assigned readings which can include selec- 
tions from Wright’s Black Boy (1993), Sumii’s A River with No Bridge 
(1990), dictionaries, and background material on burakumin printed in 
Lesson #2 below. 



Lesson 1 

Objective: To develop a definition of “bad words” and provide two 
examples. 

Procedure: Caution: Because of the nature of the subject matter, 
this lesson is best used by confident, experienced teachers who are familiar 
with the dangers of talking about controversial issues with teenagers, their 
parents, and school administrators. 

Establish workgroups of three or four students. Tell the students that 
they will read selections containing “bad words” and then will be asked to 
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use these words for specific purposes. This assignment elicits considerable 
silliness among students and generally tests the limits of classroom disci- 
pline, but anyone who works with adolescents knows how often they seek 
out and delight in the disgusting, the exotic, the painful, and the weird. 
Give the class the following list of “bad words” that are used in assigned 
readings (see references at the end). Then ask each group to discuss “bad 
words,” compose a definition, list several examples, and identify what 
kinds of prejudice each word captures. 

“Bad words’" from assigned readings: 

■ Bastard — status prejudice 

■ Eta — status prejudice (see burakumin explanation in Lesson #2 
below) 

B Nigger — race prejudice 

■ Bitch — gender prejudice 

Assignment: Ask each student to look up in the dictionary and 
write out the etymological roots of one of the words. You will need to 
supply information on eta — see Lesson #2 below. If students routinely keep 
a personal journal as part of their work for the class, ask them to respond in 
a one page journal entry to the rhyme: 

Fools’ names and foob’ faces 

Always appear in public places 

If students do not keep such a journal, this assignment can be used as 
homework. 

In a related assignment, students may be encouraged to document 
evidence of prejudice as it appears in their own communities. Students 
should record two examples of pejorative labeling that they have heard in 
their own social environments. Concurrent explorations of visual graffiti 
in art classes would make this an interesting interdisciplinary study. 
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Lesson 2 

Objective: To compare the use of language as an instrument of 
prejudice in a different cultural setting. 

Procedure: Begin the lesson by discussing new words with the class. 
Burakumin is ajapanese word meaning “village” ( buraku ) “person” (mm). 

As a group, these people have experienced a virulent form of status dis- 
crimination. Burakumin are racially and linguistically indistinguishable 
from other Japanese. They speak Japanese, they lock Japanese, but they 
have experienced a historical pattern of discrimination in their occupa- 
tions, places of residence, and marriage eligibility. Historically, they have 
been cast out ofjapanese society and considered polluted and dirty. They 
are what one writer has called an “invisible” minority, occupying subordi- 
nate positions, suffering from social disabilities, and living in segregated 
communities. Associated with burakumin are myths that they exhibit 
animal characteristics such as having a tail, growing cold to the touch at 
night, or lacking a thumb. The slang pejorative or “bad word” for 
burakumin is eta. Students who are not familiar with status discrimination in 
Japan should read the background essay, Double Binds: The Minority Child 
and the Group in Japan — The Case of the Burakumin by Finkelstein (1995) 
that is listed in the Teacher Resources Section. 

First, ask the class to define two terms from the reading selection 
given next: Komori, the community in which the burakumin characters live, 
and eta , a pejorative label for burakumin. The following questions serve as 
guides for discussion of Sumii’s book (1990), a selection of which is in- 
cluded in this lesson: 

■ What is a ghetto ? 

■ What specific characteristics of a. ghetto does the village of Komori 
manifest? 

■ To what in your experience can you relate the status of eta? 

■ What groups of people in history were forced to live in specific 
parts of a city or areas of the countryside? 
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■ What is the basis for status prejudice? Discuss as many examples as 
you can remember from your readings in history and from current- 
events. 

A River with No Bridge was published in six volumes between the 
years 1961 and 1973. The English translation by Susan Wilkerson was 
published in the United States in 1990 by Charles E. Tuttle. The work 
draws upon the experience o fbumkumin between 1908 and 1924 in Nara, 
Japan and has been a popular seller in Japan. The discrimination captured 
in the book is based on the experiences of real people, but the views of 
Koji, a main character, represent those of the author, Sue Sumii. Sumii was 
raised in the same area of Nara where the novel is set and, like Koji, was six 
years old in 1908. The following excerpt about a visit to the grandfather’s 
ho.use has been reprinted in Finkelstein and Eder (in press). 

The house at Ino was not particularly large, with no more than four 
rooms — an eight-mat, a six-mat, and two four-and-a- half-mat — but its 
high white gables looked down on the roofs of the neighboring houses, 
and the front of the rice storehouse, next to the shed, was decorated with 
a black-and-white pattern, giving it an air of prosperity. At the back of 
the house, too, there was a roofed well, with a pulley and a thick hemp 
rope, where it was possible to draw water without getting wet on rainy 
days. This well caught Koji’s fancy more than anything else at his 
grandfathers. 

As he pulled the rope, he suddenly remembered the times children 
from other villages had jeered at him — “Look at that poor good-for- 
nothing eta." “I wonder if they’re all eta at Granddad's,” he thought. 

“Master Hide, the priest’s son, said all the relatives of Komori people are 
eta too, but they look well-off here. There’s a gateway, a tiled roof, a 
storehouse, and a well with a pulley.” 

Soshichi noticed his grandson pulling the well rope and came 
hurrying out anxiously onto the veranda. “Take care you don’t slip, 

Koji,” he called. 

“I will,” replied his grandson meekly 

Soshichi did not have any favorites among his grandchildren, but he 
felt a particular concern for Seitaro and Koji because their father had 
been killed in the war; and of the two he felt closest to Koji, whose face 
was the image of Fude’s*. 
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Koji gulped some water and then, wiping his mouth, returned to his 
place by the brazier. Chie stepped down into the doma and busied herself 
preparing the afternoon meal— -in this region, it was the custom to have 
four meals a day, including one in the middle of the afternoon. Nanae 
followed her mother, hovering about the sink and stove, and Koji and his 
grandfather were left alone in the room. 

Soshichi stared at his grandson, still gripped by the feeling of unease 
and apprehension that had overtaken him a few moments before. “I 
wonder if the other children at school make him cry, too, with names 
like eta and Four-Fingers. And even if he doesn’t suffer much from that 
sort of thing now, while he’s still only a child, I expect hell have a hard 
time of it when he’s a bit older,” he thought; but he pretended to smile 
and said cheerfully, “You’ve got a bright future ahead of you, Koji, with 
your love of books.” 

Koji stared silently at one of the pages of Keizo’s storybook. 

“Don’t you think so?” pressed his grandfather. 

“What, Grandad?” 

“I said you’ve got a bright future, with vour love of books. Would 
you like to study to be a teacher?” 

Koji did not reply. 

“Don’t you want to he a teacher?” 

“Yes, I do.” 

“Well then, that’s what you’d better do.” 

“I can’t Grandad.” 

“Why not?” 

“We’re poor good-for-nothings.” A bitter little smile crossed Koji’s 
young face, at the sight of which his grandfather felt a sharp pang: it 
showed that Koji was already aware of himself as an eta . Soshichi wanted 
to weep unrestrainedly; he wanted to face the invisible enemy that was 
trying to crush them as eta and strike out at such hatred with all his 
might. At the same time, he needed no one to tell him how utterly futile 
that would be. 

“Ha, ha, ha . . . So you’re poor good-for-nothings, are you?” he 
laughed, with tears in his eyes — his way of avoiding a collision with the 
eta wall confronting him. Now sixty- five, he still did not really under- 
stand why they were what they were. Not that he was illiterate: on the 
contrary, he was one of the most educated men in Ino, with the result 
that his thinking was permeated with Confucian ideas, such as “Heaven 
cannot support two suns, nor earth two masters.” He believed that as 
long as one sun shone in the heavens, it was quite natural for there to be 
one sovereign ruler in the land, supported by a glittering array of officials 
and dignitaries. The only aspect of this scheme that he could not figure 
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out was why there were people like themselves despised as eta. Why did 
there have to be eta at all? He had resigned himself to being one by 
thinking thatit must be his karma; but this was merely a form of self- : 

deception to make life bearable. Moreover, he himself realized this; at 
heart, he did not really believe in such ideas as predestination, Stroking 
the edge of the brazier, lie said, “But there are families much poorer than 
yours, Koji. And poverty’s nothing to be afraid of. If you work hard 
enough, you’re bound to do well in the end.” 

“Were you poor to start with, Grandad?” 

“We certainly were. Not that we’re all that well-off now” 

“Yes, you are. You’ve a storehouse and a gateway, like well-to-do 
people have,’’ 

“You’ll soon be building a gateway and a storehouse at your home, too.” 
“No, wc won’t.” 

“Why not?” 

“We live in Komori.” 

Soshichi was taken back. 

“Grandad, Komori’s an eta village.” 

“Who said a thing like that?” 

“Other people do. They all do.” 

There was silence. 

“Grandad, what is an eta?” 

“I don’t know myself,” replied Soshichi, shaking his head vigorously. 

Koji looked down: he felt that his grandfather had shaken his head to 
admonish him never to mention the word eta again. 

“It’s true what Master Hide said,” he thought. “All the relatives of 
Komori people are eta. Grandad’s family, too, even though they look so 
well off. That’s why he’s angry and doesn’t want me to talk about it 
again.” (p. 99-101) 

After the students read the selection, the teacher should assign 
smaller groups of students to identify and analyze specific literary tech- 
niques. These small group sessions should address questions like the 
following: 

■ Do you think this event really happened? Why or why not? 

■ How does the author use dialogue to make the story more powerful? 

■ How does the dialogue contribute to the realism of the selection? 

■ Which part of the selection most disturbs you as a reader and why? 
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Assignment: Ask students to write two separate journal entries, if 
they keep a journal as part oF thei r class work . If they do hot keep such a 
journal, the two journal entries may be completed as assigned homework. 
Each entry should recount an instance of discrimination, either personally 
experienced or observed, that included taunts and verbal abuse. 

Students need to be reassured that these journal entries will not be 
shared with the class without permission; however Journal entries are part 
of the grade and will be read by the teacher. 

Student Evaluation 

Students can be formally evaluated on their written homework, class 
participation, and performance on the unit test. The unit test should 
provide an opportunity for students to demonstrate an increased sensitivity 
to the language of prejudice. One way to test them mighi be to require an 
essay in which students are asked to analyze one of tfye readings assigned in 
the unit, first summarizing the content of the narrative in their own words 
and next considering the way the incident dealt with in the selection is 
presented. In addition to the shock value of the language, what techniques 
does the author employ to hold a reader’s interest? The analysis should be 
at least two paragraphs, one paragraph to summarize the content and one 
to analyze the writer’s craft. 

Optional Activities 

The following questions might help guide the discussion: 

■ What cultural attitudes toward burakumin are evident in the 
selection? 

■ How is the author using language, such as pejorative labeling or 
reduction to animal status, to tell the story? 

■ Is discrimination present in all cultures? 

Since this reading is a brief except from a longer work, you may wish 
to assign or direct students to select a passage from one of the works that 
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has not been closely analyzed by the class. Teachers might also ask students 
to fin'd examples of prejudice' IhltutTeht'riewspapccs' and magazines . Tn 
such an analysis, students should be alert for examples of language that 
reflect prejudice through pejorative labeling, diminishment related to 
group membership, and reduction to animal status. Teachers could also 
assign brief research projects to explore and compare burakumin in Japan as 
illustrated in The River With No Bridge by Sumii and/or racial discrimina- 
tion in the United States as illustrated in Black Boy by Richard Wright 
( 1993 ). 

Teacher Resources 

Buraku Liberation Research Institute, 1-6, 12 Kuboyoshi, Naniwa-ku, Osaka 556, Japan. 

Finkelstein, B. (1995). Double binds: The minority child and the group in Japan — The case of the 

burakumin. College Park, MD: International Center for the Study of Education Policy 
and Human Values. 

Finkeistein, B., & Eder, E.K. (Eds.; in press). Messages of culture: Instructional materials for critical 
cultural reflection (tentative title). Yarmouth, ME: Intercultural Press. 

Shimahara, N. (1984). Toward the equality of a Japanese minority: The case of burakumin. 
Comparative Education , 20(3), 339-353. 

Sumii, S. (1990). A river with no bridge . Rutland, VT: Tuttle. 

Wright, R. (1993). Black boy (American hunger): A record of childhood and youth , NY: 

HarperPerennial. Restored text established by the Library of America. Originally 
published in 1945. 



Case Evaluation 

This selection reveals pervasive features of cultures that people find 
uncomfortable: prejudice, injustice, and life on the margins. Using the 
evaluation questions posed at the beginning of the chapter, it is easy to see 
that the creators of these lessons present quite a different concept of cul- 
ture through their instructional materials than the haiku lessons do. 

1. What concept of culture underlies the lesson? 

The authors of these lessons believe that every culture has 
values that are unexamined or not discussed. Cultural education can 
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help students examine taboo words in legitimate contexts :-such as a 
classroom. In the case of the narrative, when students read stories 
about other people, they can identify and share experiences and 
feelings, especially when those experiences are painful to think and 
talk about. In a way, the materials can give readers a voice for their 
own anger and grief and, through this medium, a way to understand 
and direct their feelings. Several assumptions about culture emerge 
from these lessons: 

■ Culture is a characteristic of groups that can be distin- 
guished by race, by gender, and especially by status. 

■ Problems of injustice, bigotry, and prejudice are universal, 
and the examination of other cultures reveals this universal 
tendency to discriminate. Because of this injustice, people 
or communities “live on the margins'’ in every culture. 

■ The study of cultures will bring about societal reform 
because when people understand cultures better they will 
eradicate the conditions that cause oppression and create a 
more just world. 

2. Is the information about the culture accurate? 

The excerpt comes from a story written first in Japanese and 
then translated into English by a competent translator and, in that 
sense, it is accurate. Since it is fiction, it is difficult to determine 
which aspects are accurate and which are not. 

3. What features of culture(s) come into view? 

The lessons reveal the “underside” ofjapanese culture and, by 
implication, all cultures. Bringing “hidden voices” of people who 
“live on the margins” to students' attention provides them with a 
more realistic portrayal of culture. For example, many Americans see 
Japan as a very homogeneous culture and are unaware of burakumin. 
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4. Does the lesson incorporate culturally unfamiliar ways of 
learning or knowing? 

By inviting discussion about taboo words in a legitimate class- 
room context, the lessons incorporate a culturally unfamiliar way of 
learning for high school students. This new way of learning and 
thinking is common among anthropologists. 

5. How are opportunities for cultural reflection structured? 

Students encounter multiple opportunities for reflecting on the 
impact of experiences with prejudice through public group discus- 
sions and private journal writing. 

6. How are learners encouraged to reflect on their own 
cultures? 

These lessons assume that people understand the influence of 
culture better when they can make personal connections with expe- 
riences of individuals through literature. Therefore the materials 
encourage students to use personal experiences with discrimination 
to interpret the assigned stories. 

Conclusion 

Both sets of materials receive high ratings from the perspective of 
good instructional design: the objectives are clear; the information about 
Japan is accurate; the activities are appropriate for and interesting to high 
school students; the skills are appropriate for English classes; and the 
method of evaluation is consonant with the stated objectives. Both cases 
cast teachers as cultural message carriers, shaping images of foreign peoples 
and cultures for the learner. Both cases presume that teachers and students 
can step outside their own cultural stereotypes and study another culture 
with some objectivity by looking at how language is used and by employ- 
ing reflective introspection. Neither case excessively praises or criticizes 
Japan. But the concept of culture that underlies the two cases, the features 
that come into view, and the structure of the lessons that help students 
“encounter” and interpret another culture are quite different. 
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The first case introduces students to a traditional literary view of 
Japanese culture and attempts to reenact an experiential approach to 
learning and entertainment for American students. The poetry topics are 
not inherently disturbing or tied to present-day problems in Japanese 
society. 

The second case dispels widely-held stereotypes about a culturally 
homogeneous Japan and concentrates on problems in modern Japanese 
society. The activities are not designed to reenact a culturally specific 
encounter but to bring out the prejudicial tendencies of all cultures. 

This analysis illustrates what evaluators can uncover about “culture 
messages” in a lesson, Neither set of instructional materials is better than 
the other from the standpoint of instructional quality, but both send 
distinctly different messages about culture that evaluators of cultural 
education programs need to be able to discern and document. 
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■ E 

J—i valuating cultural education programs from an 
economic perspective has the potential to provide 
important information to decision makers about the 
costs and outcomes of these educational initiatives. Depending on the 
available data, the level of analysis, the perspective of the evaluation, and 
the outcomes of interest, a number of questions can be answered through 
economic evaluation. Recognizing the diversity of cultural education 
programs along such dimensions as design, sources of support, and types of 
participants, this chapter seeks to provide a broad brush discussion of the 
multiple applications of economic evaluation and the variety of questions 
that can be addressed from this perspective. The goal of the chapter is to 
provide a conceptual foundation as well as a working framework to guide 
policy makers to systematically evaluate cultural education programs from 
an economic perspective. 

The first section provides general background on this genre of 
evaluation by discussing types of economic analysis, measurement of costs 
and effects, and distribution of the costs and effects across constituencies. 
The second section lays out cost and outcome templates, providing a 
flexible framework to facilitate various types of economic evaluations. The 
third section applies the templates to two hypothetical cultural education 
programs and concludes by discussing the use of this type of evaluation in 
addressing policy relevant questions about cultural education initiatives. 
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Types of Economic Analysis 

The multiple applications of cost analysis provide different types of 
valuable information to policy makers.’ Selection of the most appropriate 
type of cost analysis is dependent on the central questions guiding the 
evaluation. At the most simplistic level, a cost-feasibility analysis informs 
decision makers of the resources necessary for implementation and contin- 
ued operation of a particular program. 2 The relevant question addressed in 
a cost-feasibility analysis is “Can we afford to implement and operate this 
type of cultural education program?” Estimating cost to determine the 
cost-feasibility of a program is an essential step in deciding whether the 
initiative is a reasonable alternative to consider, given the resource base of 
the program and other supporting organizations and individuals. 

In addition to its central role in cost-feasibility analyses, information 
on program cost is the first step in further economic analyses that integrate 
data on outcomes. Chapter 4: Evaluating Program Outcomes recommended 
that evaluations of cultural education programs measure changes in partici- 
pants’ behavior in four distinct areas: (a) cultural vision, (b) knowledge, (c) 
communicative disposition, and (d) community building. Each of these 
broad categories of change is measurable through a number of indicators. 
Economic evaluations of education programs such as those discussed 
throughout this book often consider outcomes alongside costs. Two 
methods that weigh the costs against the outcomes of various alternative 
programs are cost-effectiveness and cost-benefit analyses. These ap- 
proaches shift from feasibility issues to answer questions about the value or 
worth of supporting cultural education programs. 



1 See Levin (1983) for a more detailed discussion of the various applications of cost analysis. Other useful 
descriptions of the methodology include Jamison, Klees, & Wells (1978); Popham (1988); and Rothenberg 
(1975). 

2 An example of a cost feasibility analysis is Levin & Woo (1981). 
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Cost-effectiveness analyses consider the relative costs and effects of 
several different alternatives designed to reach the same outcomes . 3 This 
type of analysis answers the question: “Should we support one program 
rather than another?” It therefore has the potential to help decision makers 
determine whether certain types of cultural education programs are, in 
fact, more effective and efficient than other alternative approaches. For 
instance, a number of differently configured cultural education programs 
may be operating with the primary goal of developing and disseminating 
written, electronic, and audiovisual curricular materials on other cultures. 
These alternative programs may vary in terms of the extensiveness of travel 
required of participants, the number of cultures addressed in the program, 
the degree to which a foreign language is taught, and the number of 
curricular areas targeted for cultural integration. Because of differences in 
program design, the cost of these multiple approaches and the level of 
effectiveness they yield can be expected to vary. 

A reasonable application of a cost-effectiveness analysis would be to 
identify the program configuration that produces the greatest level of 
effectiveness for the least cost. Using this approach, outcome measures can 
be presented in any units (e.g., number and type of curricular materials, 
the development of cultural resource centers, participation rates in cur- 
ricular advocacy efforts, or the frequency and quality of cross-cultural 
exchanges), but the outcomes and their measurement must be identical 
across the programs. For each alternative intervention, estimates of costs 
and effectiveness are integrated to form a cost-effectiveness ratio. The 
most desirable alternative, on efficiency grounds, is that with the lowest 
cost per unit of effectiveness. 

Finally, a cost-benefit analysis can be conducted for a single program 
to provide information on the degree to which the intervention is worth 
the investment . 4 Here the relevant question is “To what degree do the 



3 Examples of cost -effectiveness analyses include King f!994); Levin, Class , & Meister (1984); Mayo , 
McAnany , & Klees (1975); and Quinn, Van Mondfrans, & Worthcn (1984). 

4 Examples of cost-benefit analyses include Barnett (1985) and Stem, Dayton, Paik, & IVeisberg (1989). 
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benefits of cultural education programs outweigh the costs?” Using this 
approach, costs and benefits are presented in monetary units. For instance, 
consider an analysis of a narrowly focused cultural education program in 
which the relevant outcomes are the quantity and quality of cultural 
curricular materials produced as a result of participation in the program. 
The curricular materials produced would be assigned a monetary value, 
based perhaps on their market value. The estimated costs and benefits are 
combined in a cost-benefit ratio. To the degree that the benefits outweigh 
the costs (cost-benefit ratio is less than 1) the program is economically 
desirable. Conversely, if the costs outweigh the benefits (cost-benefit ratio 
is greater than 1) the program is not economically desirable. 

A final consideration regarding the design options of economic 
evaluations involves deciding among various levels of analysis. Cultural 
education programs can be evaluated at multiple levels. Specifically, the 
analysis can occur at a national/societal level, a regional level, a program 
level, a participant level, or a student level. The most appropriate level of 
analysis depends on the particular questions being asked in the evaluation. 
For instance, the primary funding sources may be interested in determin- 
ing the extent to which their investment is yielding adequate returns. To 
the degree that the funding source contributes widely to these programs, 
this could constitute a societal-level analysis. On the other hand, an indi- 
vidual teacher may be interested in knowing whether the time spent in the 
program will “pay off’ in the classroom; a participant level analysis such as 
this would yield important data that could be used to recruit teachers and 
education leaders to participate. Finally, decision makers at a particular 
program site may wish to determine whether one type of program is more 
cost-effective than other approaches. This would best be addressed 
through a program-level analysis. Issues of this type and level of analysis 
will be re-examined at the conclusion of this chapter. 

Measuring Costs and Outcomes 

Efforts to undertake economic evaluations of education programs are 
often aborted due to difficulties associated with measuring the costs and 
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outcomes of educational phenomena. These measurement issues pose a 
challenge to the economic evaluation of cultural education programs — 

Costs 

Every intervention or program engages a variety of resources that 
could have been used in other ways. These include hired personnel, space, 
equipment and materials, and volunteers. The total cost of the interven- 
tion incorporates the costs of all the resources devoted to achieving the 
desired outcomes of the program. The opportunity cost of a particular 
resource is the value of the next best use of that resource or the benefit 
foregone by using a resource in a particular way. 

In many cases, the opportunity cost of a resource can be represented 
by its market value. A market value is based on the price of a good or 
service in an environment of buyers and sellers. If available, the market 
value is the best estimate of the opportunity cost of a resource. As Levin 
(1983, p. 64) describes, “The market price is a measure of what must be 
sacrificed in terms of the value of other commodities to provide the 
ingredient for the intervention.” For example, the opportunity cost of a 
teacher is equal to the salary and benefits typically associated with the 
education and qualifications of that individual. 

In other cases, however, there are no explicit market values for a 
resource, so estimating opportunity cost is less straightforward. For in- 
stance, while a market value can be assigned to the time of a project direc- 
tor or teachers, it is more difficult to place a value on volunteer time. 

While the values of these resources may be difficult to pinpoint, they 
contribute to the total cost of the programs and must therefore be included 
in cost estimates. 

Outcomes 

Measuring program outcomes has been a persistent challenge in the 
field of education, and cultural education programs are no exception. 
While some aspects of effectiveness may be reasonable to quantify (e.g., the 
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number of new curricular materials or the frequency of cross-cultural 
interactions) , measuring other, important aspects of these outcomcsmay be 
far more challenging (e.g., the quality of the curricular materials and cross- 
cultural interactions). Ever more elusive is the task of assigning a mon- 
etary value to cultural education program outcomes, as required in a cost- 
benefit analysis. Consider the complexities of determining the value of 
increasing tolerance for diversity or of facilitating cross-cultural exchanges. 
In addition, the multi-faceted nature of cultural education programs 
implies that many of the initiatives are designed to effect multiple out- 
comes. Numerous complexities are associated with the necessary process 
of combining different types of outcomes so that effectiveness can be 
reduced to a single measure that is comparable across alternatives. For 
instance, weights must be attached to outcomes when more than one 
outcome is possible . 5 

Further, even the outcomes that are measurable rarely follow imme- 
diately from the implementation of an educational intervention. For 
instance, while changes in curricular offerings or the number of cultural 
materials in the school library may occur in the short term, it may be years 
before one would expect to see increases in cultural exchanges, the devel- 
opment of formal advocacy groups, or changes in the community’s per- 
ception of culture resulting from cultural education programs. 

Distribution of Costs and Outcomes across Constituencies 

Numerous stakeholders are active participants in cultural education 
programs, all exchanging investment for some expected outcome. Con- 
sider the following possibilities: Participating teachers and education 
leaders devote extensive amounts of time and energy, with the expectation 
of cultural, intellectual, and curricular returns. In addition, schools and 
districts support participants throughout the learning process with the 



5 Since this process requires some judgment on the part of the analyst, policy maker, or community, a study 
involving multiple outcomes is technically considered a cost-utility analysis (Levin, 1983). 
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intention of improving educational offerings and the school climate for 
students. Colleges and universities might serve as host institutions, provid- 
ing an infrastructure for cultural education programs with the goal of 
establishing opportunities for students and encouraging a stronger sense of 
cultural diversity. Funding agencies such as governments, foundations, 
and/or local businesses invest a considerable amount of Financial support 
with the hope of realizing higher levels of cultural understanding and 
consequent improvements in the work force and in international relations. 
This notion of multiple stakeholders implies that the costs and outcomes 
of cultural education programs are distributed across a wide range of 
individuals and organizations. Analyses that recognize this fact have the 
potential to heighten awareness of who pays and who benefits from the 
programs, yielding valuable information on both efficiency and equity. 

The efficiency contribution is straightforward. To the degree that 
inputs from a variety of sources are configured in a way that optimally 
produces desired outcomes, efficiency is achieved. Comprehensive data on 
the sources of costs and distribution of outcomes have the potential to 
inform such judgments. Implications for equity also arise from data on the 
distribution of costs and outcomes, and inequitable situations may be 
demonstrated in at least two ways: (a) the costs of the program may be 
unevenly distributed, with some groups unjustly contributing mom than 
others; and (b) the distribution of benefits may not match the distribution 
of costs such that some groups pay for, while others benefit from, the 
program. Without justification for the inequality inherent in these situa- 
tions, equity stands to be improved. 
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Design for Economic Evaluation: 

Cost and Outcome Templates 

One way to assess the costs and effects of cultural education pro- 
grams is to invoke templates that can guide policy makers through the 
process of identifying and valuing the costs and outcomes associated with 
the intervention (Rice, 1995). The cost and outcomes templates presented 
in this section provide an analytical tool to assess and integrate systemati- 
cally data on the costs and outcomes of cultural education programs. The 
templates have been designed with enough flexibility to facilitate the 
multiple types of cost analysis — cost-feasibility, cost-effectiveness, and 
cost-benefit — at various levels of analysis. These different applications of 
the templates are discussed in the final section of this chapter. 

Template for Estimating Costs 

The cost template presented in this section is intended to serve as an 
analytical device that provides a structured approach for estimating the cost 
associated with cultural education programs. The topics described below 
represent the various columns of information included in the template and 
are motivated by a series of questions intended to guide the analyst to 
unpack the costs of the resources required to support the program. An 
empty shell of the template is presented in Figure 1 and is referenced 
throughout this section. 



Figure 1: Cost Template 
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Identifying Resources to Support the Program . 

What components are included in the program? The first 
column of the template, “Program Components,” requires a listing of all 
activities and aspects of the program. In addition to the basic infrastruc- 
ture, activities comprising cultural education programs can be clustered in 
the following categories: encounters, exposure to multiple perspectives, 
formal instruction, and participation of teachers and education leaders. 

The specific activities included in any given program are likely to vary, 
depending on the scope and emphases of the program. For instance, some 
programs may foster exposure to multiple perspectives through library and 
museum films or short trips to a nearby cultural center, while others may 
require extensive travel to a foreign country. The set of activities included 
in this column is intended to guide analysts through the process of con- 
structing an organized list of the program components. The remainder of 
the analysis is highly dependent on the specification of the program as 
presented in this column. 

What resources are needed to support the program? The 

second column, “Ingredients,” requests a list of the resources to be de- 
voted to the operation of cultural education programs. The ingredients 
method (Levin, 1983) requires the specification of all the inputs that are 
intended to contribute to the desired outcomes. This specification in- 
cludes donated and volunteered resources along with resource require- 
ments that translate into expenditures. It includes all types of personnel 
resources, as well as non-personnel resources such as facilities, equipment, 
materials, and travel expenses. The list of ingredients is highly dependent 
on the program components listed in the prior column and guides the 
remainder of the template entries. Examples of ingredients that might be 
needed to support cultural education programs are listed in the second 
column of Figure 1. 

In specifying ingredients, the most time and effort should be spent 
identifying those resources that constitute the majority of the cost. For 
instance, personnel tends to account for at least two-thirds of the budget in 
typical education or social service interventions, so a great deal of attention 
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should be devoted to carefu l specification of the requirements for this 
~ category of inputs. In contrast, supplies generally comprise no more than 
five percent of the total budget. A twenty percent error in the cost of a 
category that constitutes only five percent of the total cost translates into 
only a one percent distortion (Levin, 1983). Consequently, little time 
should be spent quantifying relatively low cost items like the number of 
pencils and pens required for the program. 



Valuing the Resources 

The next task is to quantify the amount and value of each ingredient 
that is required for the program. This calculation can be tricky, especially 
when there are no explicit market values associated with the ingredients 
listed. The next five columns guide the analyst through the process of 
calculating an annual societal cost of the resources supporting the program. 

How much of each resource is required? The first column in this 
section, “Amount,” requests specification of the amount of each ingredi- 
ent listed in the previous column. The resource amounts entered in this 
column should be left in their most natural units. For instance, personnel 
resources should be recorded in terms of the number of positions needed. 
Likewise, required amounts of space, materials, equipment, and travel 
expenses should be identified in the most descriptive terms possible. The 
only restrictions are that the resources are specified on an annual basis. In 
other words, if 100 teachers participate each year, then that number should 
be entered for that ingredient. Similarly, the number of trips (round trip 
airfare), the number of per diem lodging and meal charges, the number of 
translators, and the number of group leaders should also be specified on an 
annual basis. 

What percent of annual time of each position is devoted to the 
program? Many of the individuals involved in cultural education pro- 
grams spend only a fraction of the year participating in the program. For 
instance, while some central program staff may work year-round to orga- 
nize the initiative, others, such as teachers or program translators, might 
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spend less time. The column labeled “Percent Time” is intended to ac- 
count for this partial-year commitment of some individuals. In order to 
prorate salaries to reach a figure that represents the value of the time 
devoted to program activities, this column requests information on the 
fraction of the year spent actively participating in the cultural education 
program. For instance, if teachers spend three months in the program, the 
Figure entered in this column should be .25 (three months is equal to 25 
percent of the year). Likewise, a “1” should be entered for individuals who 
work the full year planning and coordinating the programs. 

What is the annual unit value of each ingredient? The next 
column, “Unit Value,” requests a dollar value for the ingredients listed. 
The figure entered should represent the value of the next best use of the 
resource required (i.e., the opportunity cost). The market value of the 
resources should be entered, if available. In the case of personnel, this 
entry should include annual salary as well as fringe benefits, bonuses, and 
other add-ons. If a market value is not available, an approximation or 
“shadow price” for the market value can be substituted. For example, the 
shadow price of an outdated piece of technology may be estimated in 
functional terms: what would be the cost of a replacement that would 
allow the group to perform the same functions or tasks? The figure en- 
tered in this column must correspond with the units specified in the 
“Amount” column. For instance, if the number of persons is entered in 
the “Amount” column, then the annual salary for that type of position 
should be entered in the “Unit Value” column. Likewise, if a number of 
trips is specified in the “Amount” column, then the estimated value of a 
trip should be entered in the “Unit Value” column. 

What is the expected life-span of each resource? The next 
column, “Period,” requests information on the recur rence of the cost. 
Some resources are required year after year, such as salaries and benefits for 
personnel. Other resources such as equipment may be used for a number 
of years and should not simply be added into the annual cost estimate each 
year. The data in this column indicate the number of years over which 
various resources can be used. The number of years representing the 
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expected life of the resource should be entered, with recurring annual 
costs designated as “1.” 

What is the total annual societal cost of the resources? The 

final column in this section, “Total Cost,” presents a dollar figure repre- 
senting the total annual societal cost of each ingredient. This information 
should be calculated by multiplying the values in the first three columns of 
this section and then dividing by the value in the fourth. In most cases, the 
appropriate formula is: 

Total Cost = [(Amount) * (Percent Time) * (Unit Value)] / Period. 0 

The figures in this column can then be summed to derive a total 
annual societal cost estimate of the resources required to support the 
program. 



Distribution of Cost 

The burden of the various costs is often distributed across different 
individuals and organizations. For instance, in addition to the financial 
contribution of central funding sources, the cost burden of these programs 
may be shared by higher educational institutions in the U.S. and foreign 
countries, corporations, and the schools and districts of participants. 
Furthermore, the teachers and education leaders who participate in the 
services provided by a particular program also forego some opportunity 
that is associated with a cost. The costs to the various supporters of and 
participants in the program are the focus of the next set of columns of the 



6 This is an approximation of the annual cost, since, for example, it does not consider depreciation of 
equipment. 
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template. The different categories listed in the template include: school 
districts, foundations, higher education (such as a host institution), foreign 
governments, foreign businesses, and teachers. Others can be added. 

The most optimal use of these columns is to specify the exact 
amount of a particular resource that the various constituencies are ex- 
pected to shoulder. Specified in fiscal units (e.g., dollars per year), the 
entries across each row in this section should sum to equal the figure in the 
“Total Cost’’ column of that same row. Another option is to specify these 
amounts in the most natural (rather than only monetary) units. This 
descriptive information can provide insight into the types of support the 
initiative requires from different supporters of cultural education programs. 

Template for Estimating Outcomes 

Similar to the template constructed to estimate the costs of cultural 
education programs, a template can be constructed to assess the outcomes 
associated with such initiatives. Again, a series of questions are presented 
below to guide the analyst systematically through this portion of the 
analysis. A shell of this template is shown in Figure 2 7 . 



7 It should be noted that the outcomes template is used only for cost-effectiveness and cost-benefit analyses since 
the cost-feasibility analysis considers only cost, with no attention given to program outcomes. 
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Figure 2: Outcomes Template 
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Identifying Program Outcomes z ; - - 

What are the specified goals of the program? A wide array of 
goals characterize cultural education programs. As described in chapter 4, 
the intended outcomes of these types of programs fall into four general 
areas: (a) cultural vision, (b) knowledge, (c) communicative disposition, 
and (d) community building. These categories are listed in the “Identify- 
ing Outcomes” column of the template to identify broad areas that have 
the potential to prompt the evaluator to recognize and note a variety of 
program outcomes. Specific measurable outcomes, such as the number of 
cultural materials in the library, should be listed under each. 

What are the unintended outcomes of the program? Most 
interventions realize outcomes that are not necessarily specified as in- 
tended goals. Likewise, cultural education programs are likely to result in 
ancillary outcomes in addition to the four types described above. For 
instance, these programs presumably affect the students of participating 
teachers in various ways, including deepening their knowledge and under- 
standing of other cultures and increasing their cultural awareness and 
tolerance for diversity. At a more macro-level, the emphasis of the pro- 
grams on international relations may lead to improved domestic relations 
among different cultures in the U.S., increased national security, and an 
improved economic climate. These “spillover 1 effects should also be in- 
cluded in the last set of rows in the column marked “Identifying Outcomes.” 



Valuing the Outcomes 

What are the relative weights of the outcomes? Cost-effective- 
ness analysis requires that initiatives that pursue multiple outcomes assign 
weights to those outcomes so that a single measure of effectiveness that 
encompasses all intended outcomes can be calculated. Consequently he 
various expected outcomes must be ranked according to their relative 
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importance. If they are all equally valuable, equal weights of “1 ” should be 
assigned to each. If one is more valuable than the others, the weight of that 
outcome should reflect this. For instance, if the outcomes related to 
knowledge are twice as important as the other categories of outcomes, 
they should receive a weight of “2” while the others are weighted with a 
“1.” Further, if only one outcome is of interest in the analysis, that out- 
come is assigned a weight of “1” and all other outcomes are assigned 
weights of “0.” The relative weights should be indicated in the column of 
the template marked “Weight.” 

It is important to note that these weights are only relevant in the case 
of a cost-effectiveness analysis. A cost-feasibility analysis does not consider 
outcomes. A cost-benefit analysis assigns monetary values, rather than 
weights, to each of the outcomes. In addition, recall that the multiple 
programs under study in a cost-effectiveness evaluation must include the 
same set of outcomes with identical weights. 

To what degree have the outcomes been met? The focus of the 
next column labeled “Realization of Outcomes” is on reporting quantita- 
tive measures, representing the success of the program in meeting the 
outcomes specified in the first column of the template. This is an empiri- 
cal matter. A systematic study of the program’s impact on each outcome 
must be conducted to derive figures for this column. 8 This systematic 
process might involve an experimentally-designed study using random 
assignment ora quasi-experimental design (e.g., time series, non-equiva- 
lent control groups). Whatever the method, the effects of extraneous 
influences should be minimized so that program effects are isolated. 



8 While it is generally difficult (if not impossible) to present all types of outcomes in similar units, outcomes 
should be reported in such a way that they can be combined to reach a composite estimate . Weighting 
outcomes — as described in the previous section — is one approach that fosters the combination of different 
outcome measures , 
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What are the weighted outcomes resulting from the program? 

The weighted value for each outcome specified in the template is calcu- 
lated by multiplying the previous two columns: 

Weighted Outcome = Weight * Realization of Outcome . 9 

The values in this column can be summed to reach a composite 
measure of the effectiveness of the program. 



Distribution of the Outcomes 

As was the case for costs, outcomes can be distributed across different 
constituencies. For instance, participating teachers and their schools 
benefit through the development of new curricular materials. Potential 
national outcomes include improved relations between the U.S. and other 
nations and the associated economic and cultural benefits. The columns in 
the outcomes template correspond with those in the cost template. They 
include school districts, foundations, higher education, foreign govern- 
ment, foreign business, and participants. Again, additional categories can 
be added. These columns should be used to illustrate the degree to which 
program effects are distributed across various supporting individuals and 
organizations. This may be a simple matter of placing “x”s in columns of 
those stakeholders who benefit, or it may involve disaggregating the effects 
across stakeholder groups. 



9 This equation is specific for a cost-effectiveness evaluation . For a cost-benefit analysis , the relevant equation 
substitutes the monetary value of each outcome for “weight.” 
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Using the Templates: 

Merging. Cost and Outcome Data 

The cost and outcomes templates presented here can be used in 
various ways to inform decision makers about cultural education pro- 
grams. The evaluation questions are likely to vary in terms of the type of 
analysis (e.g., cost-feasibility, cost-effectiveness, cost-benefit), the level ot 
analysis, the perspective of the evaluation, and the outcomes of interest. 
Drawing on the information included in the templates, costs and out- 
comes can be integrated to address multiple types of cost-related questions. 

In this section, two hypothetical cases are presented to demonstrate a 
range of cultural education program designs and to illustrate the degree to 
which different program configurations affect costs and outcomes. Before 
presenting the cases, it is important to describe the assumptions that are 
used in the analysis. On the cost side, estimated values (representing 
hypothetical averages) are presented for a number of cost categories, as 
presented below. 



TABLE 1: Cost Assumptions in the Hypothetical Cultural Program 



Cost Category 


Estimated Value 


Annual Salaries and Benefits: 


Professional staff 


$40, 000/ year 


Support Staff 


$30 ,000/ year 


Speaker Honoraria: 


$1000 


Foreign Travel: 


Airfare 


$ 1000/trip 


Per diem (meals and lodging) 


$ 100/day 


Domestic Travel: 


Transportation (car, bus, train) 


$ 100/trip 


Per diem (meals) 


$5 0/d ay 


Office/Classroom Space 


$200/month 


Computers 


$2500 (new) 


Books/Movies 


Variable 


Materials and Supplies 


Variable 



It is important to recognize that due to the hypothetical nature of the 
cases here, these figures are illustrative; the assumptions could easily be 
altered and different numbers used. 
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On the outcomes side, an array of intended and unintended out- 
comes are identified in the areas of cultural vision, knowledge, communi- ; 
cative disposition, and community building. The hypothetical programs 
presented below both share this common set of goals and tend to place the 
greatest emphasis on outcomes in the category of cultural vision (i.e., these 
tend to be weighted more than those in the other categories). 

Hypothetical Case: Single-day Workshop 

The first cultural education program to consider is an annual single- 
day professional development workshop for teachers and school leaders in a 
particular district. This in-service day is supported by the school district to 
meet a state mandate requiring cultural training of school staff. Each year, 
the program focuses on a specific culture as well as cultural sensitivity in 
general. The 300 teachers and school leaders in the district are granted 
release time from work to attend the session at a local meeting facility. The 
program is the primary responsibility of a multicultural specialist who is a 
full-time staff person hired by the school district. The key events of the 
day include a lecture on multiculturalism and diversity by a local university 
professor, a presentation by a visiting correspondent from the specific 
culture of study, a film presenting multiple perspectives of that culture, and 
a workshop on integrating this culture into the social studies curriculum. 
The visiting correspondent is generally affiliated with the educational 
system in the other culture and is granted a leave of absence to remain in 
the U.S. for six months to serve as a consultant with the school system. 

The program costs depicted in Figure 3 are presented in terms of the 
infrastructure, encounters, exposure to multiple perspectives, formal 
instruction, and participants. Estimated values are presented in the Figure, 
based on the assumptions presented above. The total societal cost of the 
program is estimated at $147,420. As can be seen in the final set of col- 
umns, the school district shoulders the bulk of this cost ($136,400). The 
remaining cost is distributed across a number of sources: a local library pro- 
vides the film at no charge to the school district, the salary of the visiting 
teacher is split between the school district and the teacher’s home government, 
and the professor’s lecture is offered at no charge to the school district. 
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Figure 3: Application of Cost Template: Single-day Program 
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The outcomes template shown in Figure 4 includes several intended 
outcomes under each general category. In addition, several unintended 
outcomes are included in the last rows of the template. The outcomes are 
weighted so that only the intended outcomes are considered in the analysis 
(i.e., unintended outcomes are assigned a weight of “0”). Among the 
intended outcomes, the cultural vision outcomes are weighted more than 
the others. The degree to which the program meets the specified out- 
comes is shown in the third column, followed by the calculated weighted 
outcomes. As estimated here, the single-day workshop achieved 507 units 
of weighted outcome. The distribution of the outcomes is shown in the 
final set of columns, using x’s to indicate individuals or organizations likely 
to benefit. 



Figure 4: Application of Outcomes Template: Single-day Program 
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Hypothetical Case: Summer Institute - ----- 

Consider a second variation of cultural programs: a summer institute 
involving cultural immersion that is accomplished through a month-long 
trip to the culture of study. Each summer, this foundation-supported 
institute recruits 100 highly committed teachers and education leaders. 

The participants spend one month at the host university preparing for the 
trip and learning the language and the cultural mores through instruction, 
books, and films. The second month of the program is spent visiting the 
foreign country, meeting with numerous people ranging from government 
leaders to factory workers, and spending much time with educators. Upon 
returning from the trip, participants spend the final month of the institute 
at the host institution, exploring ways to mtegrate the learned concepts 
and experiences into the educational process. Activities include framing 
goals; changing curriculum, instruction, and evaluation practices; develop- 
ing networks; and conducting research. 

The costs depicted in Figure 5 include infrastructure, encounters, 
exposure to multiple perspectives, formal instruction, and participants. 

The total societal cost of the program is estimated at $1,516,400. The 
majority of this cost ($1,000,000) reflects the time that teachers invest in 
the program and is unlikely to translate into any real expenditure. The cost 
to the supporting foundation is $491,000. Other contributions include 
space at the host institution and the time of speakers in the foreign country. 

The outcomes template shown in Figure 6 includes several intended 
outcomes under each of the primary goals of cultural programs. Again, 
unintended outcomes are excluded (i.e., weights equal 0). Estimates of the 
degree to which the program meets the specified outcomes is shown in the 
third column, followed by the calculated weighted outcomes. The summer 
institute is shov.n to realize 3,460 units of outcome. The distribution of 
the outcomes is shown in the final set of columns, using “x”s to indicate 
individuals or organizations likely to benefit. 
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Figure 5: Application of Cost Template: Summer Institute 











Figure 6: Application of Outcomes Template: Summer Institute 
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Applications and Appraisal of the Templates 

In this section, the hypothetical cases presented above are used to 
illustrate the types of evaluation questions that can be addressed through 
this framework. 

QUESTION 1: 

What does the program cost and who shoulders the cost? 

The simplest type of question that can be answered using the tem- 
plate design addresses the cost-feasibility of cultural education programs. 
Here the emphasis is on estimating the value of the resources required to 
support the programs. The analysis could consider all cultural education 
programs, a particular type of program (e.g., those emphasizing language 
instruction), or a single program. Further, the evaluation questions could 
focus on costs to multiple stakeholders: foundations, government, busi- 
nesses, teachers, schools. 

Consider a cost-feasibility analysis of the two hypothetical cultural 
education programs presented above. Only a cost template is needed to 
address these questions. Figure 3 demonstrates that the total societal costs 
of the single-day workshop is equal to $147,420, and Figure 5 shows that 
the corresponding Figure for the summer institute is $1,516,400. As can be 
seen in the Final set of columns of both templates, the cost is distributed 
across various stakeholders. The school district covers the majority of the 
cost of the single-day workshop. The summer institute costs are concen- 
trated in the category of participant investment. However, these costs are 
not likely to translate into additional expenditures; participating teachers 
and educational leaders can be expected to give freely of their time to 
participate in the programs. In this case, the majority of the Financial cost- 
falls on the shoulders of the supporting foundation. 

A cost-feasibility analysis yields information that can be used to 
gauge the degree to which the cost burden of various types of cultural 
education programs can be handled. Consider a program that assumes that 
foreign school personnel will give freely of their time during American 
visits. To the degree that these individuals are either unwilling or unavail- 
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able to donate their time, the program is unlikely to succeed without a 
substitution of resources. For instance, the program could pay these 
individuals or their schools for their time, but this would increase the cost 
burden on the program budget. Extensive data on the design of the pro- 
gram, the resources required to support that design, the value of those 
resources, and the distribution of the cost burden are needed to support 
this type of analysis. 

QUESTION #2: 

Is a particular type of program more preferable than others? 

Questions that involve comparing the relative costs and effects of 
different approaches seeking to achieve similar goals are generally addressed 
through cost-effectiveness analyses. In order to address questions that 
require the comparison of costs and effectiveness of different approaches, 
data from the cost and outcomes templates must be integrated. First, the 
costs and outcomes of each alternative program must be calculated. For 
instance, Figures 3 and 4 provide the costs and outcomes for the single-day 
workshop. Next, data from the templates are integrated to form a cost- 
effectiveness ratio for each alternative. For instance, the cost-effectiveness 
ratio for the hypothetical single-day program type is 5147,420/507 or 
about $290. 10 This figure reveals the cost per unit of effectiveness of the 
single-day program. Cost-effectiveness ratios of the different program 
options are compared, and the most desirable program on economic 
grounds is that with the lowest ratio (i.e., the lowest cost per unit of 
effectiveness). 

Including all costs, the cost-effectiveness ratio for the hypothetical 
summer institute is $1,516,400/3460 or about $438." On the surface, this 
suggests that the single-day program is more cost-effective than the sum- 
mer institute. Flowever, as has been noted, a large portion of the cost of 



10 The numerator is the cost of the program as reported in Figure 3. and the denominator is the weighted 
outcome calculated in Figure 4. 

11 Here the numerator is the cost of the program as indicated in Figure 5, and the denominator is the weighted 
outcome presented in Figure 6, 
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the" summer institute is due to the extensive time required of participants, a 
cost not likely to translate into additional expenditures. When the partici- 
pant time is excluded from the cost estimates, the cost-effectiveness ratio 
of the summer institute is equal to $516,400/3460 or about $149. So an 
analysis excluding the time of participants suggests that the single-day 
program is more than twice as costly per unit of effectiveness as is the 
summer institute. 

Integrating data on costs and effects through this type of analysis can 
be extremely important for policy decisions. Information on cost alone 
provides only one piece of information. Consider the situation described 
above, where a single-day workshop is compared with a program of longer 
duration involving extensive travel: The cost templates demonstrate that 
the program requiring less participant travel translates into considerably 
lower costs. Considering cost alone would tempt policy makers to rely 
most heavily on the single-day workshop; however, when data on effec- 
tiveness are integrated into the analysis, it is clear that the effects of the 
summer institute outweigh those of the single-day workshop, making the 
former preferable on cost-effectiveness grounds. In the extreme, adopting 
an ineffective program simply because it is low cost can translate into a 
waste of valuable resources. Analyses that integrate costs and effects pro- 
vide more comprehensive data to inform decisions among alternative 
program options. 

The data required for this type of analysis include the same data 
required for cost described in the previous example, as well as extensive 
data on the effectiveness of different types of programs. It is essential that 
the multiple programs considered in a single cost-effectiveness analysis be 
pursuing similar goals and that they place similar weights on those goals. In 
other words, the first two columns in the outcomes template must be 
identical for all programs considered in a cost-effectiveness evaluation. 

QUESTION #3: 

Are cultural education programs economically worth supporting? 

A final type of question involves assessing the worth of a particular 
program, type of program, or approach in general. This involves no com- 

10 ' 
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parison among various alternatives; the focus is on whether the benefits of 
* he program outweigh the costs. Data from both the cost and outcomes 
ucmplates are needed to address such questions. However, one major 
change must be made to the outcomes template since both costs and 
outcomes must be in the same units to calculate a cost-benefit ratio. The 
“Weight” column must be changed to a “Value” column and rather than 
assign relative weights to the different outcomes, a monetary value should 
be provided for each outcome. For instance, the market value of different 
types of culturally-relevant curricular materials could be used. Costs and 
benefits are then combined in a cost-benefit ratio, and the program is 
economically desirable so long as the benefits outweigh the costs (i.e., the 
cost-benefit ratio is less than “1”). 

While this type of analysis is conceptually attractive, the data require- 
ments can be difficult to meet. Since many of the goals of cultural educa- 
tion programs are not easily translated into monetary values (i.e., market 
values are not readily accessible), cost-benefit analyses are problematic. In 
particular, assigning a value to cultural exchanges or tolerance for diversity 
poses significant problems. 



Conclusion 

This chapter provides a conceptual foundation as well as a working 
framework to guide policy makers to evaluate cultural education programs 
systematically from an economic perspective. As demonstrated throughout 
the chapter, economic-based evaluation has the potential to contribute 
important information to the decision-making arena. Questions ranging 
from the feasibility of a program to the economic worth of a program can 
be addressed using various forms of economic analysis. Perhaps most 
importantly, evaluating the costs alongside the outcomes of cultural educa- 
tion programs reveals the full range of resource commitments that teach- 
ers, students, administrators, and others have to make in order to support 
the program. This type of evaluation avoids reducing the process of cur- 
riculum transformation to a low-cost, low-investment activity. Rather, the 
approach teases out the budgetary as well as other hidden costs to reveal 
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the full cost of the program, and it weights these costs against the out- 
comes realized by the initiative. In this way, economic-based evaluation of 
cultural education programs has the potential to shed light on the worth of 
engaging in such activities and may point the way toward Realizing more 
efficient and equitable ways to reach the goals of cultural education programs. 
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Jl. ^ o single evaluation model could possibly 
serve the range of cultural education programs or 
intercultural or multicultural instructional materials or 
programs that exist. The purposes, settings, participants, and instructional 
approaches are too varied. However, all evaluations are based on measures 
to assess either outcomes or processes. This chapter emphasizes the grow- 
ing set of standardized tests and alternative assessment strategies. It offers 
evaluators a starting point for identifying appropriate, culturally sensitive 
evaluation measures. 

The chapter starts with an examination of stakeholders in the evalua- 
tion process — teachers, supervisors, administrators, and policy makers — 
and discusses ways evaluators can obtain and incorporate their perspectives 
into an evaluation plan. It then describes six traditional norm referenced 
instruments developed to specifically evaluate outcomes frequently associ- 
ated with cultural education programs, for example increased cultural 
knowledge or improved skills in cross-cultural communication. Then it 
reviews six alternative performance assessment techniques that can support 
or replace other forms of program outcome assessment. 

These alternative performance measures have the potential to im- 
prove classroom instruction in ways the standardized tests do not because 
they are directly linked to students’ learning experiences and the cultural 
contexts in which they occur. But there are dangers associated with these 
so-called “authentic” assessments and the chapter points out how these 
strategies are not immune to cultural bias. Finally it discusses several 
criteria forjudging performance strategies that highlight their potential for 
capturing the impact of a cultural education program on students. 
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Political and Educational Dimensions 
of Cultural Education Program Evaluation 

The role of an evaluator is to gather -1 ata to help inform decisions 
that are often aimed at improving programs, determining how best to 
allocate resources, or justifying a program’s continuation. Data to evaluate 
these purposes can take many forms: it can be, for example, verbal justifi- 
cations as to why the program is needed, skill attainment data, impact 
information, fiscal analysis, standardized test results, and/or case studies. 
Thus, the first steps of any evaluation are to identify the decisions to be 
made and the types of information that will aid in the decision making 
process. 

Evaluations of cultural education programs are invariably political in 
nature. Developers, advocates, and instructors all have vested professional 
interests in their outcomes. Decision makers sometimes believe they have 
a better basis for understanding the needs of students and implementing 
change than evaluators do because of their own proximity to everyday 
realities and also because they assume that the most important aspects of a 
program cannot be measured. These decision makers approach evaluation 
reports with a great deal of skepticism (King & Thompson, 1983). 

The political and psychological aspects of evaluation underscore the 
need to collect evaluation data from a variety of sources that, to the extent 
possible, address such considerations. Conflicting agendas are especially 
evident in evaluating whether or not a program has reached its goals. 
Behaviorally stated objectives have become an important tool in the design 
and conduct of evaluations. Some traditional evaluation models define the 
role of the evaluator as limited to the comparison of goals and actual 
performance (Provus, 1971 ; Tyler, 1942). A second approach is to concen- 
trate on identifying goals that are implicit in program activities (Stake, 
1975). A third approach is to focus on the intended and unintended out- 
comes of a program (Scriven, 1972). 
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In the case of cultural education programs, the goals are often vague 

and loosely constructed. Take, for example, goals for the preparation of 
curriculum materials which are often .-.rticulated in the following ways: 

■ to increase cultural self-awareness, 

■ to reduce ethnoccntrism, . 

a to improve communication skills, 

H to build understanding among individuals, 
m to build appreciation of different cultures, and 
^ to improve cross-cultural collaborative activities. 

These goals are generic, very broad, and applicable to many kinds of 
programs and their accompanying instructional materials. They rarely, 
however, reflect the objectives of decision makers. The state governor, 
school superintendent, school board, principal, curriculum supervisor, or 
teacher may have more practical objectives in mind. The governor may 
want schools to be more globally competitive. The principal or teacher 
may want to reduce mtergroup tensions and conflicts. 

Similarly, the different stakeholders will have different personal goals. 
The teacher may want to increase the chances of teaching abroad. The 
principal may want his or her name in the news as the head of a model 
multicultural school. While recognizing the goals and objectives of the 
program and instructional materials, the evaluator must also be sensitive to 
the pending decisions and personal agendas of the relevant stakeholders. 

The job of the evaluator is not just technical, but is also political and 
advisory (Fitzpatrick, 1988). Evaluators are often employed to help identify 
goals and develop strategies for accomplishing those goals. To understand 
the perspectives of the various stakeholders, the evaluator should seek 
numerous opportunities to work with them as an active participant in 
every step of the program implementation, from an original needs assess- 
ment to writing the final report. By being involved in those activities, the 
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evaluator is in a good position to translate policy questions and goals into 

-more precise questions of program, evaluation (Beswick, 1990). For ex 

ample, the evaluator may be involved in the following ways: 

■ Needs Assessment. In the university and K-12 worlds, decisions 
must be made regarding the potential benefits and trade-offs in 
providing cultural education. Evaluation personnel can help gather 
data and information to aid in these decisions and, at the same 
time, use this as an opportunity to identify the real goals of the 
organization that supports it. 

■ Establishing Goals. Evaluators need to obtain measurable goals 
from skill development to attitude change at all levels and link 
them to the original goals of the classroom, school, or district. 

m Curriculum Development. A key issue for the evaluator would 
be the degree to which cultural education is integrated into the 
school’s formal instructional program. 

a Curriculum Selection. Many organizations and commercial 
publishers have developed multicultural and intercultural materials 
for individual lessons, school-wide activities, and entire curricu- 
lum sequences in one or more subject areas. Evaluators need to 
determine how these materials are related to the desired outcomes 
of the program and the potential of the materials to achieve those 
outcomes. 

■ Advisory groups. Advisory committees made up of parents, 
representatives of community groups, teachers, and administrators 
often help to insure cultural sensitivity to the goals of a program. 
These advisory groups provide an excellent way for evaluators to 
identify stakeholder interests and biases. 




PART III: Ctofi and Benefits, Tests and Assessments 



Traditional Instruments 

While there are different program goals and different desired behav- 
iors, a common feature of cultural education programs is for the partici- 
pants to learn certain basic facts. Accordingly, one would expect there to 
be a variety of instruments related to cultural identity and cultural aware- 
ness. A search of the ERIC database, Psychlnfo, the Test Locator (http:// 
ericae2.educ.cua.edu), and the Buros Institute of Mental Measurement 
Yearbooks, however, revealed only a handful of instruments: 

■ Cultural Literacy Test (Brayfield, 1990). Designed for students 
enrolled in college level introductory sociology' classes, this instru- 
ment was used to test the hypotheses that students who have 
higher levels of cultural literacy will be successful in sociology 
classes and that white male students are more culturally literate 
than women and racial or ethnic minorities. It consists of short- 
answer questions which measure general knowledge about geogra- 
phy, important people, historical events, and current affairs. The 
test may be used by professors to assess the cultural literacy of their 
students so that they may add cultural information to the curricu- 
lum as needed (description from the Test Locator). 

■ Multiethnic Awareness Survey (1977). This is a 40-minute, 
100-item test that measures factual knowledge about the contribu- 
tions of minorities to the past and present growth of the United 
States. Items reflect awareness of deeds of Black, Irish, Italian, 
Jewish, Asian, Latin and Native Americans (description from the 
Test Locator). 

■ Measures of Global Understanding (Educational Testing 
Service, 1981). Designed for use in a study commissioned by the 
Council on Learning. This 90-minute test measures knowledge, 
attitudes, perceptions, and interests of college students related to 
other cultures and the political events, national policies, and other 
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factors affecting these cultures. The test also elicits students' 

' descriptions of their own foreign languag^e profrcieiicy (description 
from the Test Locator). 

u Multicultural Counseling Inventory (MCI; Sodowsky, Taffe, 
Gutkm, & Wise, 1994). This 40-item self-report instrument was 
designed to operationalize the constructs of multicultural counsel- 
ing competencies proposed in the literature. Subscales are coun- 
seling skills, multicultural awareness, multicultural counseling 
knowledge, and multicultural counseling relationship. Items tend 
to focus on behaviors rather than attitudes. Quality validity evi- 
dence is presented. 

H Multicultural Counseling Awareness Scale-Form B (MCA.S- 
B; Ponterotto, Sanchez, & Magids, 1991). The developers of this 
42-item self-report inventory derived their items from the litera- 
ture and expert panel review. Subscales include a Knowledge/ 

Skills scale and an Awareness scale. Included is a interesting Social 
Desirability cluster which provides a measure of assessment quality. 

■ Multicultural Awareness-Knowledge Skills Survey (MAKSS; 
D'Andrea, Daniels, & Heck, 1991). A 60-item self-report instru- 
ment containing three scales: Awareness, Knowledge, and Sldlls. 
The author’s primary claim to validity is based on item construc- 
tion. Construct validation v/as conducted on a small sample 1 . 

The lack of publicly available instruments indicates that many evalua- 
tions are relying on tests that come with textbooks and locally developed 
instruments. While these instruments may be responsive to local needs, 



K An excellent review of (he last three measures can be found in Pope- Davis and Dings (1995). 
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there is a need for (a) more instruments with good technical documents- 
don, and (b) more sharing of instruments across programs'.' 



Additional Indicators 

While extremely important, participant cultural awareness ispniy 
one of a wide number of desired possible outcomes for an evaluator to 
consider. Recent research in cognitive sciences supports the concept of 
intelligence as a cultural construction. Gardner and Hatch (1989) define 
intelligence as “the capacity to solve problems or to fashion products that 
are valued in one or more cultural settings” (p. 5). Using this definition, 
Gardner developed the concept of multiple intelligences, that is, skills and 
behaviors that individuals exhibit in different patterns and degrees 
(Brualdi, 1996). These seven intelligences im lude: logical-mathematical, 
linguistic, spatial, musical, bodily-kinesthctic, and personal (including 
interpersonal, or the ability to understand the feelings of others, and 
intrapersonal, or the ability to understand one's own feelings and motiva- 
tions). 

If individuals have different patterns of skills and behaviors, tradi- 
tional instruments may not properly capture this diversity, especially when 
individuals exhibit patterns based on cultural differences that arc unfamil- 
iar to the test maker. For instance, DeRoche (1987), Gardner (1983), and 
others argue that relying on an array of achievement tests overemphasizes 
some intelligences and disregards others. Evaluators can gather additional 
information to capture some of these differences by looking for objectives 
that would measure a student's: 

H performance in events outside of school such as community 
groups; 

M skill in using information from school within a particular cultural 
context such as knowing who to ask for directions in an unfamiliar 
location and how to phrase the question; 
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E facility in demonstrating several different ways to_tell a story, 

present an argument, describe an event, participate in a classroom 
discussion, or complete a small group project; or 

M ability to demonstrate social values such as tolerance, respect, and 
fairness to others. 



Some Performance Assessment Techniques 

Information on any knowledge, attitude, or activity resulting from 
cross-cultural education can be used to help inform decision making. 
While we want unbiased, psychomctrically sound, and gcncralizablc 
information, not all data must meet these requirements. Rather, placed in 
context, even the softest information can be quite useful. 

Consider, for example, a cross-cultural education program designed 
for elementary school teachers. One level of data might be that 85% of the 
participating teachers showed large improvements on the Multicultural 
Counseling Inventory. Another level of data could be case studies showing 
impact. For example, one teacher has established E-mail interviews with 
students from another country. Another has reviewed all of her handouts 
to be sure they take a more global viewpoint. The writing in the school 
newsletter has showed a marked increase in pluralistic values. While the 
first level is objective and gcncralizablc, the second level is more informa- 
tive. Evaluations should contain a mix of both information levels. Readers 
might consult chapter 4 for other performance-based outcomes. 

The following list identifies a range of information sources that can 
be used to help support cross-cultural education evaluations. 

E Products. Improved products are often the desired end-product 
of training activities. They require a broad range of competencies, 
are often interdisciplinary in focus, and require initiative and 
creativiry. As an assessment device, trainedjudges should score 
each product against agreed-upon standards of excellence. 
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M Group projects, Group projects enable a number of students to 
work together on aoromplcx^ problem that requires planning; 
research, internal discussion, and group presentation. This tech- 
nique is particularly attractive because it facilitates cooperation, 
reinforces a valued outcome, and is sensitive to the orientation of 
group members. 

m Interviews/Oral presentations. Interviews and oral presenta- 
tions allow individuals to verbalize their knowledge and are more 
likely to elicit informative responses than open-ended, written 
questions. The 1969 and 1976 National Assessments of Educa- 
tional Progress (NAEP) Citizenship Assessments, for example, 
used many interview questions. An obvious example of oral 
assessment occurs in the foreign languages: fluency can be assessed 
only by hearing the student sneak. 

■ Constructed-response questions. Constructed-response 
questions require students to produce their own answers rather 
than select from an array of possible answers (as with multiple- 
choice items). A constructed-response question may have just 
one correct answer, or it may be more open ended and allow a 
range of responses. 

■ Writing samples. Writing samples have long been used to assess a 
student’s understanding of a subject through a written description, 
analysis, explanation, or summary. Essays can demonstrate how 
well a student uses facts in context and structures a coherent 
discussion. Answering essay questions effectively requires critical 
thinking, analysis, and synthesis. 

Pratt (1972) proposed a useful tool for capturing the cultural 
orientation of written material. The basic procedure is to (a) 
define material to be analyzed, (b) define subjects of interest, (c) 
prepare scoring sheets and word lists, (d) scan materials for refer- 
ences to the subjects of interest, (e) list judgmental terms, (f) 
compare judgmental terms to the word list, and (g) calculate the 
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Evaluation Coefficient (the ratio of related positive terms to the 
'"'sum' of positive and' negative terms expressed as a percentage). See 
Sapp and Skelton (1987) and Garcia et al. (1976) for applications of 
Pratt’s Evaluation Coefficient. 

■ Portfolios. Portfolios are usually files or folders that contain 
collections of materials. As an assessment tool, the trainee is asked 
to furnish a broad portrait of individual performance, assembled 
over time. As trainees put together their portfolios, they must 
evaluate their own work. Student drafts, revisions, works in 
progress, and final papers are included to show development. 



Issues and Warnings 

Performance assessments, like standardized tests, have their short- 
comings. Due to the fact that they rely heavily on judgment, performance 
tests are susceptible to the traditional concerns about the use of raters. To 
fairly score students, Stiggins (1987, p. 33) states it is critical that the 
performance ratings reflect the examinee’s true capabilities and are not a function of 
the perceptions and biases of the persons evaluating the performance. In addition, 
because there arc fewer data points, there are also serious psychometric 
issues related to the use of performance assessments. These issues include: 

■ Social desire. The examinee may select answers that reflect his or 
her perception regarding desirable responses. 

H Self-deception. Respondents’ answers may reflect an incorrect 
perception regarding how they would behave under certain 
circumstances. 

tt Halo effect. Impressions that an evaluator has formed about an 
individual can influence his or her impression of that person along 
other dimensions. Nisbett and Wilson (1977), for example, made 
a videotape of a professor acting in a friendly manner and another 
of the same professor acting in an arrogant manner. Students who 
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watched the friendly professor tape rated the professor more 
■favorably on other traits like physical appearance^nd mannerisms: - ' 

m Stereotyping. Impressions that an evaluator has formed about a 
group of individuals can alter his or her impressions of a member 
of that group. A principal, for example, might find an individual 
mathematics teacher to be precise due his perception that all 
mathematics teachers arc precise rather than based on the 
individual's actual performance. 

■ Perception differences. The past experiences and viewpoints of 
an evaluator can affect what he or she focuses on and the way he or 
she interprets behavior. In a classic study in the business literature, 
Dearborn and Simon (1958) asked business executives to identify 
the major problem described in a detailed case study. The execu- 
tives tended to view the problem in terms of their own depart- 
mental functions. 

m Leniency error. When a rater does not have enough knowledge 
to make an objective rating, he or she might either give the benefit 
of the doubt to the person being rated or systematically rate them 
downward. 

■ Equity issues. Are the assessment tasks sensitive to the linguistic 
and cultural diversity of the trainees? Performance tasks can place 
a significant linguistic demand on students. 

m Psychometric issues. Many members of the measurement 
community have urged caution with the use of performance 
assessment due to psychometric concerns. Koretz, Madaus, 
Haertel, and Beaton testified before Congress that the psychomet- 
rics of alternative assessment has not been developed sufficiently 
to warrant its use as the basis for a high-stakes national assessment 
system Linn, Baker, and Dunbar (1991) also questioned tlie 
inferences to be drawn from some high-order assessment items. 
Koretz et al. noted that scores in one high-profile state assessment 
varied from rater to rater. 
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While there are a large number of potential threats to the validity of _ 
alternative assessments, the threats do not necessarily limit their usefulness. 
In evaluation we are interested in making generalizations at the program 
level, not at the individual level. Evaluation data is not used to make 
judgments about individuals. The real question is whether the instrument 
will provide a good group estimate. Data that is invalid or unreliable for 
some individuals, may yield very useful and accurate information when 
aggregated to the group level. Thus, the usual concerns for high reliability 
and validity do not apply. Rather the threats to validity should be recog- 
nized and efforts taken to minimize their impact, and the evaluator should 
ask whether the group level inferences arc reasonable. 



Multisource Appraisal 

A common approach in the corporate world for evaluating employ- 
ees is to pool information from others who have direct knowledge of the 
employee’s job performance, even though this is subjective data (Black, 
Gregcrsen, &: Mendenhall, 1992). The same principle of multisourcc 
appraisal can be applied to teachers who participate in cultural education 
programs or who use multicultural instructional materials. People from 
whom evaluators could seek information about a teacher’s understanding 
about cultures include: 

■ (from students) the teacher’s success in interpersonal communica- 
tion skills including strategies for promoting inclusion of all 
students in classroom activities; 

■ (from other teachers) the teacher’s team building efforts and other 
methods of working to build communities and eliminate status 
differences that perpetuate inequities of the larger society in the 
school and classroom; 

M (from the principal) the teacher’s understanding of students’ 
cultures including language proficiencies; knowledge of how to 
evaluate the language and cultural demands of classroom tasks; and 
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ss (from the individual teacher) his or her own ability to describe the 
"influence of culture in shaping his or her ~6\vri ^ practice 

in the classroom; opportunities the individual takes to develop a 
deeper understanding about particular cultural communities 
represented in the school or curriculum. 

This wide range of data provides a rich description of the teacher's 
performance that can help in evaluations. 



Criteria forjudging 
Performance Assessments 

Linn, Baker, and Dunbar ( 1991 ) proposed several criteria forjudging 
the quality of performance assessment data. While their criteria were 
developed in the context of high-stakes assessment, they arc fully appli- 
cable in an evaluation setting. Their most important criteria are (as sum- 
marized by Miles, 1093): 

■ Fairness. Have we fairly selected the test components? Do our 
scoring practices fairly reflect students’ capabilities? How are we 
going to use and interpret the results? Gaps exist between students 
because of differences in their familiarity with and exposure to the 
test components and in their motivation to perform and learn. 
Miller-Jones (1989) suggests that we use “functionally equivalent 
tasks specific to the culture and instructional context of the 
individual being assessed” (p. 362). 

■ Transfer and gencralizability. How far do skills in one ai ea 
transfer to another? What generalizations can \vc make from the 
test results? Measuring the degree to which skills transfer within a 
performance-based assessment is heavily dependent upon the task 
being performed. 

■ Cognitive complexity. Does the test require students to use 
higher-order thinking skills to solve and analyze problems instead 
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of memorizing facts and solving well structured, decontex- 
- tualized problems? We should not assume, for example, that all - 
performance activities require the use of complex, cognitive 
processes. Judge the cognitive complexity by analyzing the task. 
Then, factor in the student’s familiarity with the problem and the 
student’s approach to solving the problem. 

N Content quality. Is the content of the assessment consistent with 
the current understanding in the field? Will the content stand the 
test of time? Most important, the content should be worth the 
student’s and the rater’s time and effort. To assure the quality of 
the content, we may want experts in the subject matter to review 
both the tasks that the student performs and the overall design of 
the assessment itself. This is especially important in domains like 
cultural studies where the content is elusive and difficult for all but 
content specialises to discover and reveal. 

■ Content coverage. Does the assessment adequately cover the 
subject matter? As Collins, Hawkins, and Frederiksen (1990) 
noted, both students and teachers tend to undcr-emphasize infor- 
mation not covered in the assessment. 

m Meaningfulness* Does the assessment give students meaningful 
problems? Do the students gain worthwhile educational experi- 
ences? To find out if the assessment is meaningful, analyze the 
performance tasks or ask the students and teachers what they 
thought of the tasks. Although analyzing the tasks can provide 
relevant input, finding out the understandings and reactions of the 
students and teachers provides more valuable, systematic informa- 
tion on how meaningful the assessment is. 

m Cost and efficiency. Can we justify the cost of these more labor- 
intensive assessments? 

These criteria can be applied to any instrument the evaluator is 
considering as an outcome measure. 
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Summary 

After identifying the decisions to be made, the evaluator needs to 
identify the types of data that will help inform those decisions. The evalua- 
tor can then turn to identify appropriate culturally sensitive instruments. 
While there arc several instruments that appear to be applicable over a 
wide range of situations, the evaluator may prefer a locally developed 
instrument. There is a wide variety of formats that can be employed, 
including products, group projects, interviews/oral presentations, con- 
structed-response questions, writing samples, and portfolios. Regardless of 
the format, there are several threats to validity that must be considered, 
especially if the evaluation is to depend on raters or judges. Some of these 
threats are self-deception, halo effect, stereotyping, perception differences, 
leniency error, equity issues, and psychometric issues. The chapter ends 
with a discussion of criteria that can be used to evaluate assessment instru- 
ments: social desire, fairness, transfer and generalizability, cognitive com- 
plexity, content quality, content coverage, meaningfulness, cost, and 
efficiency. 

It is an exciting rime for the evaluation field. Twenty or thirty years 
ago, emphasis was on commercially purchased standardized tests. Today, 
clients and the evaluation field appreciate the advantage of locally devel- 
oped instruments — they have the potential to provide the most relevant 
and useful data. 
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Today’s cultural education movement emphasizes in-depth study 
of foreign and domestic cultures as an effective method of 
understanding alternative ways of thinking, feeling, and interacting. 
It is only through cultural awareness that educators and business 
people can be effective in today's diverse global society. 
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Discovering Culture in Education will help program planners, 
teachers, and evaluators as they organize approaches to 
diversity, frame educational goafs, choose aprons alternative 
curriculum contents, evaluate instructional materials, formulate 
culturally sensitive forms of evaluation, and develop ways to 
monitor, evaluate, refine, and document program effectiveness. 
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